[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:58:46 dist-test-slave-39f2 krb5kdc[21164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:58:46 dist-test-slave-39f2 krb5kdc[21164](info): set up 2 sockets
Sep 27 04:58:46 dist-test-slave-39f2 krb5kdc[21164](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 04:58:47.292630 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:58:47 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:58:47.468344 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39275
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:47.977411 21177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:47.977823 21177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:47.978052 21177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:47.987869 21177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:58:47.988081 21177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:47.988222 21177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:47.988384 21177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:58:47.988503 21177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:58:48.001080 21177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:48.002740 21177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:48.004308 21177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:48.021454 21182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:48.022334 21185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:48.022346 21183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:48.024158 21177 server_base.cc:1034] running on GCE node
I20260927 04:58:48.025213 21177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:48.026952 21177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:48.028277 21177 hybrid_clock.cc:648] HybridClock initialized: now 1790485128028173 us; error 82 us; skew 500 ppm
I20260927 04:58:48.028765 21177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:48 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485128, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:48.033299 21177 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:58:48.035502 21177 webserver.cc:460] Webserver started at http://127.20.169.126:34403/ using document root <none> and password file <none>
I20260927 04:58:48.036384 21177 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:48.036602 21177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:48.037109 21177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:48.041138 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "67c1b5c16d2245059379094aff522979"
format_stamp: "Formatted at 2026-09-27 04:58:48 on dist-test-slave-39f2"
server_key: "38787655eab7b814e39a2546aee39a2e"
server_key_iv: "2dc442e56626d6bf773a8dd14a6fd7da"
server_key_version: "encryptionkey@0"
I20260927 04:58:48.042155 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "67c1b5c16d2245059379094aff522979"
format_stamp: "Formatted at 2026-09-27 04:58:48 on dist-test-slave-39f2"
server_key: "38787655eab7b814e39a2546aee39a2e"
server_key_iv: "2dc442e56626d6bf773a8dd14a6fd7da"
server_key_version: "encryptionkey@0"
I20260927 04:58:48.049295 21177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 04:58:48.054100 21192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:48.055615 21177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 04:58:48.055874 21177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "67c1b5c16d2245059379094aff522979"
format_stamp: "Formatted at 2026-09-27 04:58:48 on dist-test-slave-39f2"
server_key: "38787655eab7b814e39a2546aee39a2e"
server_key_iv: "2dc442e56626d6bf773a8dd14a6fd7da"
server_key_version: "encryptionkey@0"
I20260927 04:58:48.056221 21177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:48.080631 21177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:48.085342 21177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:48.085650 21177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:48.112260 21177 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:39275
I20260927 04:58:48.112325 21244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:39275 every 8 connection(s)
I20260927 04:58:48.113938 21177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:58:48.119011 21245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:48.119738 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21177
I20260927 04:58:48.120146 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:58:48.123205 21157 external_mini_cluster.cc:1428] Setting key 12525c7fc09d923ec9b00f6c84c9b004
I20260927 04:58:48.137821 21245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979: Bootstrap starting.
I20260927 04:58:48.142109 21245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:48.143919 21245 log.cc:824] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:48.148516 21245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979: No bootstrap required, opened a new log
I20260927 04:58:48.160180 21245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c1b5c16d2245059379094aff522979" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39275 } }
I20260927 04:58:48.160913 21245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:48.161123 21245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c1b5c16d2245059379094aff522979, State: Initialized, Role: FOLLOWER
I20260927 04:58:48.162077 21245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c1b5c16d2245059379094aff522979" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39275 } }
I20260927 04:58:48.162427 21245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:58:48.162703 21245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:58:48.162981 21245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 04:58:48 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:48.167906 21245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c1b5c16d2245059379094aff522979" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39275 } }
I20260927 04:58:48.168675 21245 leader_election.cc:302] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67c1b5c16d2245059379094aff522979; no voters:
I20260927 04:58:48.169984 21245 leader_election.cc:288] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:58:48.170245 21250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:48.172742 21250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [term 1 LEADER]: Becoming Leader. State: Replica: 67c1b5c16d2245059379094aff522979, State: Running, Role: LEADER
I20260927 04:58:48.173583 21250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c1b5c16d2245059379094aff522979" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39275 } }
I20260927 04:58:48.173774 21245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:58:48.177929 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67c1b5c16d2245059379094aff522979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c1b5c16d2245059379094aff522979" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39275 } } }
I20260927 04:58:48.178694 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [sys.catalog]: This master's current role is: LEADER
I20260927 04:58:48.182421 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:48.129214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.20.169.126:39275)
0927 04:58:48.132896 (+ 3682us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:48.132911 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:48.132985 (+ 74us) server_negotiation.cc:407] Connection header received
0927 04:58:48.146359 (+ 13374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:48.146404 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:48.146918 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:48.147405 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:48.151587 (+ 4182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:48.153569 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:48.155760 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:48.156716 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:48.157050 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:48.166615 (+ 9565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:48.166625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:48.166641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:48.166705 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:48.172843 (+ 6138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:48.174289 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:48.174299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:48.174307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:48.174643 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:48.175529 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:48.175536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:48.175540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:48.176728 (+ 1188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:48.177229 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:48.180105 (+ 2876us) server_negotiation.cc:299] Negotiation successful
0927 04:58:48.180614 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3300,"thread_start_us":2823,"threads_started":1}
I20260927 04:58:48.188527 21255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:58:48.188269 21252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c1b5c16d2245059379094aff522979. Latest consensus state: current_term: 1 leader_uuid: "67c1b5c16d2245059379094aff522979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c1b5c16d2245059379094aff522979" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39275 } } }
I20260927 04:58:48.188784 21252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979 [sys.catalog]: This master's current role is: LEADER
I20260927 04:58:48.194729 21255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:58:48.204558 21255 catalog_manager.cc:1347] Generated new cluster ID: fd516a89aa2940a9916b9a1dfbce258d
I20260927 04:58:48.204680 21255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:58:48.234710 21255 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:58:48.237308 21255 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:58:48.263056 21255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67c1b5c16d2245059379094aff522979: Generated new TSK 0
I20260927 04:58:48.264061 21255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:58:48.321398 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39275
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:48.868393 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:48.868721 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:48.868963 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:48.879247 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:48.879487 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:48.879719 21271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:58:48.892306 21271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:48.893873 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:48.895468 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:48.912608 21276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:48.913043 21277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:48.913344 21279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:48.918563 21271 server_base.cc:1034] running on GCE node
I20260927 04:58:48.919394 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:48.920452 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:48.921734 21271 hybrid_clock.cc:648] HybridClock initialized: now 1790485128921689 us; error 58 us; skew 500 ppm
I20260927 04:58:48.922120 21271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:48 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485128, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:48.925556 21271 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:58:48.927105 21271 webserver.cc:460] Webserver started at http://127.20.169.65:45233/ using document root <none> and password file <none>
I20260927 04:58:48.927810 21271 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:48.927907 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:48.928148 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:48.931109 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "fe52945fea1b431689c72e36442d9050"
format_stamp: "Formatted at 2026-09-27 04:58:48 on dist-test-slave-39f2"
server_key: "b5a4bf820ef2ad79b9bdccb005941bf6"
server_key_iv: "92b8e443ebc0a81cf278d6ea27c8b7be"
server_key_version: "encryptionkey@0"
I20260927 04:58:48.931883 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "fe52945fea1b431689c72e36442d9050"
format_stamp: "Formatted at 2026-09-27 04:58:48 on dist-test-slave-39f2"
server_key: "b5a4bf820ef2ad79b9bdccb005941bf6"
server_key_iv: "92b8e443ebc0a81cf278d6ea27c8b7be"
server_key_version: "encryptionkey@0"
I20260927 04:58:48.937139 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 04:58:48.940723 21286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:48.942235 21271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:58:48.942400 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "fe52945fea1b431689c72e36442d9050"
format_stamp: "Formatted at 2026-09-27 04:58:48 on dist-test-slave-39f2"
server_key: "b5a4bf820ef2ad79b9bdccb005941bf6"
server_key_iv: "92b8e443ebc0a81cf278d6ea27c8b7be"
server_key_version: "encryptionkey@0"
I20260927 04:58:48.942570 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:48.968752 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:48.973133 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:48.973470 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:48.974553 21271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:58:48.976186 21271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:48.976337 21271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:48.976569 21271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:48.976691 21271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:49.053481 21271 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:37007
I20260927 04:58:49.053644 21399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:37007 every 8 connection(s)
I20260927 04:58:49.055073 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:58:49.056962 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21271
I20260927 04:58:49.057385 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:58:49.058636 21157 external_mini_cluster.cc:1428] Setting key 9f8e95a824d887539397e69a2fbe31dc
Sep 27 04:58:49 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485128, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 04:58:49.083452 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:49.059383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:46875 (local address 127.20.169.126:39275)
0927 04:58:49.060544 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:49.060552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:49.062099 (+ 1547us) server_negotiation.cc:407] Connection header received
0927 04:58:49.064764 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:49.064770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:49.064888 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:49.065023 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:49.067009 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.067933 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.070002 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.070527 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.070662 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:49.075742 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:49.075751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:49.075759 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:49.075802 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:49.079824 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:49.080694 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:49.080704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:49.080706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:49.080929 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:49.081443 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:49.081449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:49.081452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:49.081789 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:49.081941 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:49.082725 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 04:58:49.082971 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":711,"threads_started":1}
I20260927 04:58:49.084769 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:49.060447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39275 (local address 127.20.169.65:46875)
0927 04:58:49.061900 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:49.061990 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:49.064497 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:49.065347 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:49.065356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:49.065988 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:49.066808 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:49.066827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.068120 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.068124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:49.069837 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:49.069867 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.070845 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.070854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:49.071165 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:49.071983 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:49.072005 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:49.075489 (+ 3484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:49.079991 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:49.080001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:49.080018 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:49.080560 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:49.081042 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:49.081047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:49.081050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:49.081300 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:49.081937 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:49.081945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:49.082295 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 04:58:49.082581 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 04:58:49.083236 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":792,"threads_started":1}
I20260927 04:58:49.086784 21400 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39275
I20260927 04:58:49.087183 21400 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:49.088176 21400 heartbeater.cc:511] Master 127.20.169.126:39275 requested a full tablet report, sending...
I20260927 04:58:49.090855 21209 ts_manager.cc:194] Registered new tserver with Master: fe52945fea1b431689c72e36442d9050 (127.20.169.65:37007)
I20260927 04:58:49.092484 21209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:46875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:58:49.104888 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39275
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:49.641222 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:49.641507 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:49.641603 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:49.648600 21406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:49.648694 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:49.648828 21406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:58:49.658310 21406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:49.659644 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:49.661091 21406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:49.678179 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:49.679203 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:49.679219 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:49.680763 21406 server_base.cc:1034] running on GCE node
I20260927 04:58:49.681685 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:49.682880 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:49.684171 21406 hybrid_clock.cc:648] HybridClock initialized: now 1790485129684117 us; error 62 us; skew 500 ppm
I20260927 04:58:49.684585 21406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:49 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485129, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:49.687888 21406 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:58:49.689656 21406 webserver.cc:460] Webserver started at http://127.20.169.66:46557/ using document root <none> and password file <none>
I20260927 04:58:49.690395 21406 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:49.690497 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:49.690745 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:49.693572 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "a1f66905d42f411ab4973f56b7bb3853"
format_stamp: "Formatted at 2026-09-27 04:58:49 on dist-test-slave-39f2"
server_key: "adc3a11b66cb04899fc9a21a1ba2980a"
server_key_iv: "a592a70dbb1a9668bac2e812929a4c75"
server_key_version: "encryptionkey@0"
I20260927 04:58:49.694301 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "a1f66905d42f411ab4973f56b7bb3853"
format_stamp: "Formatted at 2026-09-27 04:58:49 on dist-test-slave-39f2"
server_key: "adc3a11b66cb04899fc9a21a1ba2980a"
server_key_iv: "a592a70dbb1a9668bac2e812929a4c75"
server_key_version: "encryptionkey@0"
I20260927 04:58:49.699744 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 04:58:49.703553 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:49.704931 21406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 04:58:49.705056 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "a1f66905d42f411ab4973f56b7bb3853"
format_stamp: "Formatted at 2026-09-27 04:58:49 on dist-test-slave-39f2"
server_key: "adc3a11b66cb04899fc9a21a1ba2980a"
server_key_iv: "a592a70dbb1a9668bac2e812929a4c75"
server_key_version: "encryptionkey@0"
I20260927 04:58:49.705219 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:49.716920 21406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:49.720660 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:49.720954 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:49.721817 21406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:58:49.723063 21406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:49.723133 21406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:49.723212 21406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:49.723248 21406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:49.798867 21406 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:34715
I20260927 04:58:49.799019 21533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:34715 every 8 connection(s)
I20260927 04:58:49.800446 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:58:49.808555 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21406
I20260927 04:58:49.808889 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:58:49.809652 21157 external_mini_cluster.cc:1428] Setting key 87e98b314ce12ea3b5e388303188b220
Sep 27 04:58:49 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485129, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:49.831152 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:49.803679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:57837 (local address 127.20.169.126:39275)
0927 04:58:49.804444 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:49.804448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:49.805573 (+ 1125us) server_negotiation.cc:407] Connection header received
0927 04:58:49.808155 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:49.808161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:49.808243 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:49.808377 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:49.811170 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.812190 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.815473 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.816334 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.816445 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:49.821521 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:49.821530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:49.821538 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:49.821583 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:49.826861 (+ 5278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:49.827760 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:49.827770 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:49.827775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:49.828026 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:49.828626 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:49.828633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:49.828636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:49.829175 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:49.829378 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:49.830252 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 04:58:49.830523 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":500,"threads_started":1}
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:58:49.832147 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:49.804252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39275 (local address 127.20.169.66:57837)
0927 04:58:49.805410 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:49.805451 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:49.807804 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:49.808736 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:49.808749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:49.809941 (+ 1192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:49.810943 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:49.810991 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.812339 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.812344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:49.815266 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:49.815286 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:49.816651 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:49.816661 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:49.817106 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:49.818048 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:49.818072 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:49.821212 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:49.827029 (+ 5817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:49.827037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:49.827051 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:49.827595 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:49.828173 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:49.828176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:49.828178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:49.828455 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:49.829314 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:49.829320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:49.829726 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 04:58:49.830080 (+ 354us) client_negotiation.cc:240] Negotiation successful
0927 04:58:49.830705 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":617,"threads_started":1}
I20260927 04:58:49.833813 21534 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 04:58:49.834324 21534 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:49.835641 21534 heartbeater.cc:511] Master 127.20.169.126:39275 requested a full tablet report, sending...
I20260927 04:58:49.837554 21209 ts_manager.cc:194] Registered new tserver with Master: a1f66905d42f411ab4973f56b7bb3853 (127.20.169.66:34715)
I20260927 04:58:49.838508 21209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:57837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:58:49.860464 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39275
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:58:50.096108 21400 heartbeater.cc:503] Master 127.20.169.126:39275 was elected leader, sending a full tablet report...
W20260927 04:58:50.366735 21540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:50.367085 21540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:50.367327 21540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:50.376597 21540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:50.376798 21540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:50.377084 21540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:58:50.390448 21540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:50.391913 21540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:50.393507 21540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:50.411087 21545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:50.411260 21548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:50.411818 21546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:50.412654 21540 server_base.cc:1034] running on GCE node
I20260927 04:58:50.413331 21540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:50.414310 21540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:50.415683 21540 hybrid_clock.cc:648] HybridClock initialized: now 1790485130415579 us; error 50 us; skew 500 ppm
I20260927 04:58:50.416122 21540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:50.420704 21540 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:58:50.422824 21540 webserver.cc:460] Webserver started at http://127.20.169.67:42881/ using document root <none> and password file <none>
I20260927 04:58:50.423658 21540 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:50.423915 21540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:50.424338 21540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:50.428170 21540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "cd8ed8c66a2b4efda004cf109b06de1d"
format_stamp: "Formatted at 2026-09-27 04:58:50 on dist-test-slave-39f2"
server_key: "7e7108c372fc257641fd2c029a5cb4c6"
server_key_iv: "4a1916b2ae6daf8262a7eab4403d1214"
server_key_version: "encryptionkey@0"
I20260927 04:58:50.429133 21540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "cd8ed8c66a2b4efda004cf109b06de1d"
format_stamp: "Formatted at 2026-09-27 04:58:50 on dist-test-slave-39f2"
server_key: "7e7108c372fc257641fd2c029a5cb4c6"
server_key_iv: "4a1916b2ae6daf8262a7eab4403d1214"
server_key_version: "encryptionkey@0"
I20260927 04:58:50.435791 21540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 04:58:50.440438 21555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:50.442013 21540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 04:58:50.442260 21540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "cd8ed8c66a2b4efda004cf109b06de1d"
format_stamp: "Formatted at 2026-09-27 04:58:50 on dist-test-slave-39f2"
server_key: "7e7108c372fc257641fd2c029a5cb4c6"
server_key_iv: "4a1916b2ae6daf8262a7eab4403d1214"
server_key_version: "encryptionkey@0"
I20260927 04:58:50.442584 21540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:50.475971 21540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:50.482179 21540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:50.482889 21540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:50.484604 21540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:58:50.486665 21540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:50.486819 21540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:50.487078 21540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:50.487243 21540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:50.548645 21540 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:38789
I20260927 04:58:50.548962 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:38789 every 8 connection(s)
I20260927 04:58:50.550436 21540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:58:50.554237 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21540
I20260927 04:58:50.554610 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:58:50.555352 21157 external_mini_cluster.cc:1428] Setting key 545b22e958d60f5c6bd70628b0769eec
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:50.577899 21673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.554206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:58541 (local address 127.20.169.126:39275)
0927 04:58:50.555047 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:50.555054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:50.556618 (+ 1564us) server_negotiation.cc:407] Connection header received
0927 04:58:50.559587 (+ 2969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:50.559594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:50.559729 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:50.559871 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:50.561969 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.562965 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.564914 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.565470 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.565596 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.570107 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:50.570113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:50.570116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:50.570148 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:50.574373 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.575162 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.575167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.575169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.575534 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.576005 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.576008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.576011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.576367 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:50.576479 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:50.577337 (+ 858us) server_negotiation.cc:299] Negotiation successful
0927 04:58:50.577495 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":514,"threads_started":1}
I20260927 04:58:50.578909 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.555028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39275 (local address 127.20.169.67:58541)
0927 04:58:50.556424 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:50.556479 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:50.559329 (+ 2850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:50.560152 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:50.560160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:50.560952 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:50.561773 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:50.561792 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.563151 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.563156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:50.564695 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:50.564707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.565743 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.565748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:50.566026 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.567011 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:50.567051 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:50.569843 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:50.574489 (+ 4646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:50.574496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:50.574508 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:50.575018 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:50.575639 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:50.575642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:50.575645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:50.575905 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:50.576478 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:50.576484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:50.576878 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 04:58:50.577133 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 04:58:50.577549 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":691,"threads_started":1}
I20260927 04:58:50.580628 21671 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39275
I20260927 04:58:50.581030 21671 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:50.581945 21671 heartbeater.cc:511] Master 127.20.169.126:39275 requested a full tablet report, sending...
I20260927 04:58:50.583606 21209 ts_manager.cc:194] Registered new tserver with Master: cd8ed8c66a2b4efda004cf109b06de1d (127.20.169.67:38789)
I20260927 04:58:50.585049 21209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:58541
I20260927 04:58:50.585930 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:58:50.638998 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.618953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.20.169.65:37007)
0927 04:58:50.620072 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:50.620078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:50.620104 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:58:50.620163 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:50.620169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:50.620411 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:50.620578 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:50.622437 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.623723 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.625548 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.626285 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.626391 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.632032 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:50.632043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:50.632057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:50.632092 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:50.635881 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.636638 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.636645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.636647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.636859 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.637276 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.637280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.637282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.637651 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:50.637978 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:50.638264 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 04:58:50.638475 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":693,"threads_started":1}
W20260927 04:58:50.640471 21314 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:58:50.690147 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.673315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.20.169.65:37007)
0927 04:58:50.673576 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:50.673582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:50.673776 (+ 194us) server_negotiation.cc:407] Connection header received
0927 04:58:50.673940 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:50.673943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:50.674075 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:50.674181 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:50.675973 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.676744 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.678307 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.678734 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.678831 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.683769 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:50.683774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:50.683777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:50.683808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:50.686992 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.687848 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.687852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.687854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.688051 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.688439 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.688443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.688445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.688733 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:50.689049 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:50.689378 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 04:58:50.689572 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 04:58:50.691861 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21271
I20260927 04:58:50.709079 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.693036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.20.169.65:37007)
0927 04:58:50.693429 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:50.693437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:50.693757 (+ 320us) server_negotiation.cc:407] Connection header received
0927 04:58:50.693942 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:50.693948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:50.694011 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:50.694109 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:50.695760 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.696666 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.698363 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.698990 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.699128 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.702195 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:50.702200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:50.702203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:50.702231 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:50.705612 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.706573 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.706579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.706582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.706788 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.707267 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.707274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.707277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.707598 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:50.707933 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:50.708212 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 04:58:50.708452 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 04:58:50.709996 21395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:50.794533 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21271
I20260927 04:58:50.812306 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21406
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:58:50.829103 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.813679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.20.169.66:34715)
0927 04:58:50.814285 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:50.814291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:50.814316 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:58:50.814382 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:50.814388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:50.814583 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:50.814708 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:50.815982 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.817148 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.818351 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.818874 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.818980 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.823083 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:50.823094 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:50.823109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:50.823166 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:50.826304 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.826994 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.827007 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.827014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.827187 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.827611 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.827615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.827617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.827969 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:50.828190 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:50.828441 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 04:58:50.828626 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":319,"threads_started":1}
I20260927 04:58:50.830551 21529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:50.912705 21534 heartbeater.cc:503] Master 127.20.169.126:39275 was elected leader, sending a full tablet report...
I20260927 04:58:50.915496 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21406
W20260927 04:58:50.940366 21194 connection.cc:535] server connection from 127.20.169.66:57837 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:58:50.940502 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21540
Sep 27 04:58:50 dist-test-slave-39f2 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:58:50.956746 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:50.941648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.20.169.67:38789)
0927 04:58:50.942136 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:50.942144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:50.942172 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:58:50.942231 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:50.942237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:50.942471 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:50.942601 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:50.943777 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.944768 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.946028 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:50.946567 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:50.946671 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:50.951113 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:50.951123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:50.951137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:50.951175 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:50.954105 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.954765 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.954772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.954783 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.954936 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:50.955282 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:50.955285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:50.955287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:50.955616 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:50.955831 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:50.956160 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 04:58:50.956349 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260927 04:58:50.958146 21666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:51.041363 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21540
I20260927 04:58:51.058954 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21177
I20260927 04:58:51.060191 21240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:51.161815 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21177
2026-09-27T04:58:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5005 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:58:51 dist-test-slave-39f2 krb5kdc[21693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:58:51 dist-test-slave-39f2 krb5kdc[21693](info): set up 2 sockets
Sep 27 04:58:51 dist-test-slave-39f2 krb5kdc[21693](info): commencing operation
krb5kdc: starting...
W20260927 04:58:52.249163 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:58:52 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:58:52.432052 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38557
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:52.750022 21706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:52.750317 21706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:52.750412 21706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:52.757598 21706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:58:52.757784 21706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:52.757869 21706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:52.757974 21706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:58:52.758046 21706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:58:52.768687 21706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:52.769915 21706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:52.771152 21706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:52.783963 21712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:52.784660 21711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:52.785669 21714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:52.785969 21706 server_base.cc:1034] running on GCE node
I20260927 04:58:52.786850 21706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:52.788242 21706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:52.789451 21706 hybrid_clock.cc:648] HybridClock initialized: now 1790485132789406 us; error 32 us; skew 500 ppm
I20260927 04:58:52.789811 21706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:52 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485132, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:52.798418 21706 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:58:52.800045 21706 webserver.cc:460] Webserver started at http://127.20.169.126:37057/ using document root <none> and password file <none>
I20260927 04:58:52.800760 21706 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:52.800931 21706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:52.801280 21706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:52.804174 21706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "aa49200f210b4b98a5d82370d6bee1d9"
format_stamp: "Formatted at 2026-09-27 04:58:52 on dist-test-slave-39f2"
server_key: "e0821ed860b6baf840c65a712063327f"
server_key_iv: "325f06c31fb0885d2e4473549e8073ab"
server_key_version: "encryptionkey@0"
I20260927 04:58:52.805001 21706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "aa49200f210b4b98a5d82370d6bee1d9"
format_stamp: "Formatted at 2026-09-27 04:58:52 on dist-test-slave-39f2"
server_key: "e0821ed860b6baf840c65a712063327f"
server_key_iv: "325f06c31fb0885d2e4473549e8073ab"
server_key_version: "encryptionkey@0"
I20260927 04:58:52.810273 21706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 04:58:52.813678 21721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:52.815016 21706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 04:58:52.815200 21706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "aa49200f210b4b98a5d82370d6bee1d9"
format_stamp: "Formatted at 2026-09-27 04:58:52 on dist-test-slave-39f2"
server_key: "e0821ed860b6baf840c65a712063327f"
server_key_iv: "325f06c31fb0885d2e4473549e8073ab"
server_key_version: "encryptionkey@0"
I20260927 04:58:52.815404 21706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:52.835431 21706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:52.839126 21706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:52.839397 21706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:52.857570 21706 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:38557
I20260927 04:58:52.857611 21773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:38557 every 8 connection(s)
I20260927 04:58:52.858865 21706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:58:52.862486 21774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:52.868306 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21706
I20260927 04:58:52.868556 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:58:52.869223 21157 external_mini_cluster.cc:1428] Setting key caa834f24a9c90d26aec705b0a491855
I20260927 04:58:52.873457 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9: Bootstrap starting.
I20260927 04:58:52.877336 21774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:52.878444 21774 log.cc:824] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:52.881595 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9: No bootstrap required, opened a new log
Sep 27 04:58:52 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:52.890251 21774 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa49200f210b4b98a5d82370d6bee1d9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38557 } }
I20260927 04:58:52.890630 21774 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:52.890762 21774 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa49200f210b4b98a5d82370d6bee1d9, State: Initialized, Role: FOLLOWER
I20260927 04:58:52.891474 21774 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa49200f210b4b98a5d82370d6bee1d9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38557 } }
I20260927 04:58:52.891772 21774 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:58:52.891937 21774 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:58:52.892138 21774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:52.893348 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:52.871264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.20.169.126:38557)
0927 04:58:52.872098 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:52.872109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:52.872148 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:58:52.873934 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:52.873959 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:52.874325 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:52.874669 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:52.876615 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:52.878090 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:52.879724 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:52.880350 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:52.880517 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:52.885284 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:52.885292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:52.885308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:52.885350 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:52.888630 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:52.889449 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:52.889457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:52.889463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:52.889653 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:52.890057 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:52.890064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:52.890084 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:52.890711 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:52.891064 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:52.891577 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 04:58:52.892012 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":369,"threads_started":1}
I20260927 04:58:52.895153 21774 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa49200f210b4b98a5d82370d6bee1d9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38557 } }
I20260927 04:58:52.895684 21774 leader_election.cc:302] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa49200f210b4b98a5d82370d6bee1d9; no voters:
I20260927 04:58:52.896330 21774 leader_election.cc:288] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:58:52.896430 21779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:52.897255 21779 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [term 1 LEADER]: Becoming Leader. State: Replica: aa49200f210b4b98a5d82370d6bee1d9, State: Running, Role: LEADER
I20260927 04:58:52.897848 21779 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa49200f210b4b98a5d82370d6bee1d9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38557 } }
I20260927 04:58:52.898530 21774 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:58:52.900477 21780 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa49200f210b4b98a5d82370d6bee1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa49200f210b4b98a5d82370d6bee1d9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38557 } } }
I20260927 04:58:52.900960 21780 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:58:52.901859 21787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:58:52.901809 21781 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa49200f210b4b98a5d82370d6bee1d9. Latest consensus state: current_term: 1 leader_uuid: "aa49200f210b4b98a5d82370d6bee1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa49200f210b4b98a5d82370d6bee1d9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38557 } } }
I20260927 04:58:52.902269 21781 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:58:52.909482 21787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:58:52.920677 21787 catalog_manager.cc:1347] Generated new cluster ID: 83eea8d3472d47df9ecda6397c3e4ce2
I20260927 04:58:52.920790 21787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:58:52.938156 21787 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:58:52.939428 21787 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:58:52.954671 21787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa49200f210b4b98a5d82370d6bee1d9: Generated new TSK 0
I20260927 04:58:52.955633 21787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:58:53.015398 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38557
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:53.339030 21800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:53.339284 21800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:53.339462 21800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:53.346552 21800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:53.346647 21800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:53.346863 21800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:58:53.356444 21800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:53.357748 21800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:53.359084 21800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:53.372135 21805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:53.372175 21806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:53.372092 21808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:53.373072 21800 server_base.cc:1034] running on GCE node
I20260927 04:58:53.373598 21800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:53.374441 21800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:53.375636 21800 hybrid_clock.cc:648] HybridClock initialized: now 1790485133375579 us; error 56 us; skew 500 ppm
I20260927 04:58:53.375962 21800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:53 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485133, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:53.379197 21800 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:58:53.380643 21800 webserver.cc:460] Webserver started at http://127.20.169.65:39183/ using document root <none> and password file <none>
I20260927 04:58:53.381366 21800 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:53.381508 21800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:53.381820 21800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:53.385126 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "5b53d55455ee48c39f304929899667f5"
format_stamp: "Formatted at 2026-09-27 04:58:53 on dist-test-slave-39f2"
server_key: "cbe743e2f5eafb417f24ceadce56a98e"
server_key_iv: "8cedce058eb93446e7a348f84eca37a2"
server_key_version: "encryptionkey@0"
I20260927 04:58:53.386003 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "5b53d55455ee48c39f304929899667f5"
format_stamp: "Formatted at 2026-09-27 04:58:53 on dist-test-slave-39f2"
server_key: "cbe743e2f5eafb417f24ceadce56a98e"
server_key_iv: "8cedce058eb93446e7a348f84eca37a2"
server_key_version: "encryptionkey@0"
I20260927 04:58:53.391404 21800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:58:53.395222 21815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:53.396555 21800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:58:53.396759 21800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "5b53d55455ee48c39f304929899667f5"
format_stamp: "Formatted at 2026-09-27 04:58:53 on dist-test-slave-39f2"
server_key: "cbe743e2f5eafb417f24ceadce56a98e"
server_key_iv: "8cedce058eb93446e7a348f84eca37a2"
server_key_version: "encryptionkey@0"
I20260927 04:58:53.397023 21800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:53.427920 21800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:53.432802 21800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:53.433168 21800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:53.434238 21800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:58:53.435647 21800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:53.435758 21800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:53.435927 21800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:53.436019 21800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:53.472172 21800 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:45149
I20260927 04:58:53.472229 21928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:45149 every 8 connection(s)
I20260927 04:58:53.473655 21800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:58:53.484016 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21800
I20260927 04:58:53.484252 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:58:53.484935 21157 external_mini_cluster.cc:1428] Setting key e1cd69c8dfc0d16b550ee487e47c83a4
Sep 27 04:58:53 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485133, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:53.494827 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:53.476379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:35549 (local address 127.20.169.126:38557)
0927 04:58:53.476932 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:53.476939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:53.478017 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 04:58:53.479853 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:53.479857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:53.479926 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:53.480028 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:53.481651 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.482417 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.484025 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.484569 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.484715 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:53.488664 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:53.488670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:53.488674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:53.488708 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:53.491511 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:53.492273 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:53.492283 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:53.492285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:53.492450 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:53.492936 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:53.492940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:53.492942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:53.493206 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:53.493380 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:53.494220 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 04:58:53.494470 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":244,"threads_started":1}
I20260927 04:58:53.495810 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:53.477088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38557 (local address 127.20.169.65:35549)
0927 04:58:53.477827 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:53.477884 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:53.479565 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:53.480233 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:53.480242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:53.480803 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:53.481476 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:53.481493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.482584 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.482588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:53.483871 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:53.483883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.484799 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.484841 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:53.485139 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:53.485793 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:53.485816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:53.488292 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:53.491675 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:53.491685 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:53.491703 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:53.492139 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:53.492579 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:53.492584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:53.492587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:53.492780 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:53.493345 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:53.493353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:53.493835 (+ 482us) client_negotiation.cc:769] Sending connection context
0927 04:58:53.494100 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 04:58:53.494629 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260927 04:58:53.497479 21929 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38557
I20260927 04:58:53.497826 21929 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:53.498757 21929 heartbeater.cc:511] Master 127.20.169.126:38557 requested a full tablet report, sending...
I20260927 04:58:53.501446 21738 ts_manager.cc:194] Registered new tserver with Master: 5b53d55455ee48c39f304929899667f5 (127.20.169.65:45149)
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 04:58:53.503429 21738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:35549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:58:53.534578 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38557
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:53.848407 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:53.848690 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:53.848788 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:53.855795 21935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:53.855893 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:53.856004 21935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:58:53.865373 21935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:53.866731 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:53.868536 21935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:53.883972 21943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:53.884192 21941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:53.884064 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:53.885419 21935 server_base.cc:1034] running on GCE node
I20260927 04:58:53.885973 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:53.886818 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:53.888048 21935 hybrid_clock.cc:648] HybridClock initialized: now 1790485133887990 us; error 51 us; skew 500 ppm
I20260927 04:58:53.888368 21935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:53 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485133, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:53.891515 21935 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:58:53.893127 21935 webserver.cc:460] Webserver started at http://127.20.169.66:39483/ using document root <none> and password file <none>
I20260927 04:58:53.893810 21935 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:53.893961 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:53.894280 21935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:53.897511 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "99c5ab644e494a2d84b91f64a1e3e730"
format_stamp: "Formatted at 2026-09-27 04:58:53 on dist-test-slave-39f2"
server_key: "6371f8c55cad519a13a02ed148f2099a"
server_key_iv: "da97844174ef0a6c0df7bfbc5e4ad692"
server_key_version: "encryptionkey@0"
I20260927 04:58:53.898317 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "99c5ab644e494a2d84b91f64a1e3e730"
format_stamp: "Formatted at 2026-09-27 04:58:53 on dist-test-slave-39f2"
server_key: "6371f8c55cad519a13a02ed148f2099a"
server_key_iv: "da97844174ef0a6c0df7bfbc5e4ad692"
server_key_version: "encryptionkey@0"
I20260927 04:58:53.904218 21935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 04:58:53.907789 21950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:53.909227 21935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:58:53.909399 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "99c5ab644e494a2d84b91f64a1e3e730"
format_stamp: "Formatted at 2026-09-27 04:58:53 on dist-test-slave-39f2"
server_key: "6371f8c55cad519a13a02ed148f2099a"
server_key_iv: "da97844174ef0a6c0df7bfbc5e4ad692"
server_key_version: "encryptionkey@0"
I20260927 04:58:53.909600 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:53.937628 21935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:53.941987 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:53.942284 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:53.943269 21935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:58:53.944595 21935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:53.944698 21935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:53.944859 21935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:53.944952 21935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 04:58:53.977555 21935 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:41811
I20260927 04:58:53.977588 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:41811 every 8 connection(s)
I20260927 04:58:53.978936 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:58:53.982342 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 21935
I20260927 04:58:53.982564 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:58:53.983245 21157 external_mini_cluster.cc:1428] Setting key 495bd2ef76877bb0398a04fb62d823b0
Sep 27 04:58:53 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485133, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:53.999428 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:53.981742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:49651 (local address 127.20.169.126:38557)
0927 04:58:53.981930 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:53.981935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:53.983518 (+ 1583us) server_negotiation.cc:407] Connection header received
0927 04:58:53.985877 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:53.985882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:53.985974 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:53.986080 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:53.987546 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.988251 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.989805 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.990265 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.990395 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:53.993825 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:53.993830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:53.993833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:53.993863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:53.996465 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:53.997240 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:53.997249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:53.997252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:53.997373 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:53.997808 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:53.997813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:53.997818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:53.998062 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:53.998210 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:53.998861 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 04:58:53.999079 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:58:54.000450 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:53.982538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38557 (local address 127.20.169.66:49651)
0927 04:58:53.983335 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:53.983384 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:53.985619 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:53.986305 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:53.986313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:53.986803 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:53.987361 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:53.987378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.988408 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.988420 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:53.989671 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:53.989682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:53.990498 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:53.990505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:53.990721 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:53.991374 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:53.991394 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:53.993616 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:53.996616 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:53.996624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:53.996639 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:53.997116 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:53.997503 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:53.997508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:53.997511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:53.997691 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:53.998196 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:53.998204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:53.998510 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 04:58:53.998750 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 04:58:53.999271 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
I20260927 04:58:54.002106 22064 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38557
I20260927 04:58:54.002568 22064 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 04:58:54.003516 22064 heartbeater.cc:511] Master 127.20.169.126:38557 requested a full tablet report, sending...
I20260927 04:58:54.005271 21738 ts_manager.cc:194] Registered new tserver with Master: 99c5ab644e494a2d84b91f64a1e3e730 (127.20.169.66:41811)
I20260927 04:58:54.006521 21738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:49651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:58:54.026913 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38557
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:54.375710 22069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:54.375959 22069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:54.376108 22069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:54.383430 22069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:54.383564 22069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:54.383739 22069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:58:54.393679 22069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:54.394996 22069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:54.396476 22069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:54.416860 22075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:54.416900 22074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:54.417537 22077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:54.423993 22069 server_base.cc:1034] running on GCE node
I20260927 04:58:54.424788 22069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:54.426074 22069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:54.427330 22069 hybrid_clock.cc:648] HybridClock initialized: now 1790485134427251 us; error 63 us; skew 500 ppm
I20260927 04:58:54.427759 22069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:54 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485134, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:54.432126 22069 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:58:54.434108 22069 webserver.cc:460] Webserver started at http://127.20.169.67:36193/ using document root <none> and password file <none>
I20260927 04:58:54.435106 22069 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:54.435253 22069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:54.435633 22069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:54.440227 22069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "5343fe4ecc094a80a4d03b6079fe6008"
format_stamp: "Formatted at 2026-09-27 04:58:54 on dist-test-slave-39f2"
server_key: "b23a0d9768a74190a0b82113c7acbbf5"
server_key_iv: "a45a111575af08304ad192756501d9c1"
server_key_version: "encryptionkey@0"
I20260927 04:58:54.441373 22069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "5343fe4ecc094a80a4d03b6079fe6008"
format_stamp: "Formatted at 2026-09-27 04:58:54 on dist-test-slave-39f2"
server_key: "b23a0d9768a74190a0b82113c7acbbf5"
server_key_iv: "a45a111575af08304ad192756501d9c1"
server_key_version: "encryptionkey@0"
I20260927 04:58:54.448720 22069 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 04:58:54.453603 22084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:54.455245 22069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:58:54.455410 22069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "5343fe4ecc094a80a4d03b6079fe6008"
format_stamp: "Formatted at 2026-09-27 04:58:54 on dist-test-slave-39f2"
server_key: "b23a0d9768a74190a0b82113c7acbbf5"
server_key_iv: "a45a111575af08304ad192756501d9c1"
server_key_version: "encryptionkey@0"
I20260927 04:58:54.455644 22069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:54.480573 22069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:54.484982 22069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:54.485263 22069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:54.486330 22069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:58:54.487756 22069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:54.487826 22069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:54.487991 22069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:54.488040 22069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:54.506551 21929 heartbeater.cc:503] Master 127.20.169.126:38557 was elected leader, sending a full tablet report...
I20260927 04:58:54.535825 22069 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:45129
I20260927 04:58:54.535879 22196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:45129 every 8 connection(s)
I20260927 04:58:54.537415 22069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:58:54.547215 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 22069
I20260927 04:58:54.547487 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:58:54.548107 21157 external_mini_cluster.cc:1428] Setting key 981027bd428d6bba8a920b39ed8691df
Sep 27 04:58:54 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485134, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:54.565231 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.541278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:57653 (local address 127.20.169.126:38557)
0927 04:58:54.542047 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.542052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.543774 (+ 1722us) server_negotiation.cc:407] Connection header received
0927 04:58:54.546456 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.546462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.546553 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.546669 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:54.548731 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.550458 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.552909 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.553476 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.553617 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.558015 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:54.558022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:54.558029 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:54.558069 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:54.561332 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:54.562129 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:54.562137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:54.562142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:54.562310 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:54.562904 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:54.562911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:54.562914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:54.563347 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:54.563570 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.564473 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.564705 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":523,"threads_started":1}
I20260927 04:58:54.566344 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.542364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38557 (local address 127.20.169.67:57653)
0927 04:58:54.543612 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:54.543656 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:54.546173 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:54.546943 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:54.546951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:54.547739 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:54.548554 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.548575 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.550638 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.550642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.552728 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.552742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.553811 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.553821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.554085 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.554918 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:54.554964 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:54.557751 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:54.561473 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:54.561483 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:54.561500 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:54.561986 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:54.562427 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:54.562432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:54.562436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:54.562740 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:54.563568 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:54.563578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:54.564045 (+ 467us) client_negotiation.cc:769] Sending connection context
0927 04:58:54.564363 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 04:58:54.564918 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":635,"threads_started":1}
I20260927 04:58:54.567891 22197 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38557
I20260927 04:58:54.568228 22197 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:54.569307 22197 heartbeater.cc:511] Master 127.20.169.126:38557 requested a full tablet report, sending...
I20260927 04:58:54.570977 21738 ts_manager.cc:194] Registered new tserver with Master: 5343fe4ecc094a80a4d03b6079fe6008 (127.20.169.67:45129)
I20260927 04:58:54.571868 21738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:57653
I20260927 04:58:54.577579 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:58:54.602385 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.585467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.20.169.126:38557)
0927 04:58:54.585672 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.585679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.586576 (+ 897us) server_negotiation.cc:407] Connection header received
0927 04:58:54.586718 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.586722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.586814 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.586894 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:54.589164 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.590055 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.591761 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.592309 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.592427 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.595782 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:54.595787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:54.595791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:54.595819 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:54.599280 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:54.600109 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:54.600113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:54.600115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:54.600281 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:54.600686 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:54.600689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:54.600691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:54.601107 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:54.601406 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.601717 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.601911 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 04:58:54.609556 21738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:58:54.619143 21738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:58:54.654174 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.634530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.20.169.65:45149)
0927 04:58:54.636250 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.636261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.636600 (+ 339us) server_negotiation.cc:407] Connection header received
0927 04:58:54.637531 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.637541 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.637892 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.638393 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:54.640508 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.642961 (+ 2453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.649539 (+ 6578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.652208 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.652451 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.652850 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.653234 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.653504 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":1923,"threads_started":1}
I20260927 04:58:54.654323 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.633930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45149 (local address 127.0.0.1:40460)
0927 04:58:54.636280 (+ 2350us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:54.636357 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:54.637233 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:54.638183 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:54.638194 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:54.638271 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:54.640251 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.640265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.644265 (+ 4000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.644273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.649242 (+ 4969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.649264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.652594 (+ 3330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.652604 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.652954 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.653030 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 04:58:54.653165 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 04:58:54.653525 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"thread_start_us":3543,"threads_started":1}
I20260927 04:58:54.659343 21863 tablet_service.cc:1467] Processing CreateTablet for tablet 20edaaf4b9e24ead9001ac1d1be99f40 (DEFAULT_TABLE table=test-table [id=2979db2980a84a4aa0e1669933027d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:54.661190 21863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20edaaf4b9e24ead9001ac1d1be99f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:54.664297 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.636370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:41811 (local address 127.0.0.1:51774)
0927 04:58:54.643903 (+ 7533us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:54.643928 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:54.644083 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:54.653132 (+ 9049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:54.653138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:54.653167 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:54.654322 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.654338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.656766 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.656783 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.660138 (+ 3355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.660152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.663144 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.663151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.663421 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.663454 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 04:58:54.663545 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:58:54.663705 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7326,"thread_start_us":7014,"threads_started":1}
I20260927 04:58:54.665020 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.636643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:45129 (local address 127.0.0.1:56064)
0927 04:58:54.651250 (+ 14607us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:54.651281 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:54.651442 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:54.653794 (+ 2352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:54.653799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:54.653827 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:54.654964 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.654989 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.657751 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.657756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.661062 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.661079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.663656 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.663663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.663924 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.663958 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 04:58:54.664063 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 04:58:54.664201 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14435,"thread_start_us":14877,"threads_started":1}
I20260927 04:58:54.665079 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.636218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.20.169.67:45129)
0927 04:58:54.652664 (+ 16446us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.652672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.652725 (+ 53us) server_negotiation.cc:407] Connection header received
0927 04:58:54.652872 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.652880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.653077 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.653294 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:54.655274 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.657554 (+ 2280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.661316 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.663375 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.663529 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.663859 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.664109 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.664386 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16208,"thread_start_us":15866,"threads_started":1}
I20260927 04:58:54.665529 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.634766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.20.169.66:41811)
0927 04:58:54.651971 (+ 17205us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.651984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.652022 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:58:54.652176 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.652183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.652455 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.652729 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:54.654673 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.656641 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.660466 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.662809 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.662984 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.663194 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.664752 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.664999 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17016,"thread_start_us":16770,"threads_started":1}
I20260927 04:58:54.670017 21998 tablet_service.cc:1467] Processing CreateTablet for tablet 20edaaf4b9e24ead9001ac1d1be99f40 (DEFAULT_TABLE table=test-table [id=2979db2980a84a4aa0e1669933027d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:54.670017 22132 tablet_service.cc:1467] Processing CreateTablet for tablet 20edaaf4b9e24ead9001ac1d1be99f40 (DEFAULT_TABLE table=test-table [id=2979db2980a84a4aa0e1669933027d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:54.671881 22132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20edaaf4b9e24ead9001ac1d1be99f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:54.671917 21998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20edaaf4b9e24ead9001ac1d1be99f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:54.677829 22217 tablet_bootstrap.cc:492] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: Bootstrap starting.
I20260927 04:58:54.681974 22217 tablet_bootstrap.cc:654] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:54.683262 22217 log.cc:824] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:54.687321 22217 tablet_bootstrap.cc:492] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: No bootstrap required, opened a new log
I20260927 04:58:54.687592 22217 ts_tablet_manager.cc:1397] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260927 04:58:54.689710 22219 tablet_bootstrap.cc:492] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: Bootstrap starting.
I20260927 04:58:54.690835 22220 tablet_bootstrap.cc:492] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: Bootstrap starting.
I20260927 04:58:54.696785 22219 tablet_bootstrap.cc:654] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:54.697201 22220 tablet_bootstrap.cc:654] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:54.698954 22220 log.cc:824] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:54.699064 22219 log.cc:824] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:54.702883 22220 tablet_bootstrap.cc:492] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: No bootstrap required, opened a new log
I20260927 04:58:54.703192 22220 ts_tablet_manager.cc:1397] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260927 04:58:54.703742 22217 raft_consensus.cc:348] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.704506 22217 raft_consensus.cc:374] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:54.704694 22217 raft_consensus.cc:729] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b53d55455ee48c39f304929899667f5, State: Initialized, Role: FOLLOWER
I20260927 04:58:54.705752 22217 consensus_queue.cc:260] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.709973 22217 ts_tablet_manager.cc:1428] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: Time spent starting tablet: real 0.022s user 0.009s sys 0.014s
I20260927 04:58:54.713764 22219 tablet_bootstrap.cc:492] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: No bootstrap required, opened a new log
I20260927 04:58:54.714131 22219 ts_tablet_manager.cc:1397] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.020s
I20260927 04:58:54.717120 22220 raft_consensus.cc:348] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.717595 22220 raft_consensus.cc:374] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:54.717732 22220 raft_consensus.cc:729] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5343fe4ecc094a80a4d03b6079fe6008, State: Initialized, Role: FOLLOWER
I20260927 04:58:54.718484 22220 consensus_queue.cc:260] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.720285 22197 heartbeater.cc:503] Master 127.20.169.126:38557 was elected leader, sending a full tablet report...
I20260927 04:58:54.721055 22220 ts_tablet_manager.cc:1428] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: Time spent starting tablet: real 0.018s user 0.018s sys 0.001s
W20260927 04:58:54.726166 21930 tablet.cc:2369] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:58:54.726147 22219 raft_consensus.cc:348] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.726673 22219 raft_consensus.cc:374] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:54.726852 22219 raft_consensus.cc:729] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c5ab644e494a2d84b91f64a1e3e730, State: Initialized, Role: FOLLOWER
I20260927 04:58:54.727576 22219 consensus_queue.cc:260] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.728991 22064 heartbeater.cc:503] Master 127.20.169.126:38557 was elected leader, sending a full tablet report...
I20260927 04:58:54.729786 22219 ts_tablet_manager.cc:1428] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
W20260927 04:58:54.731071 22065 tablet.cc:2369] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:58:54.790445 22199 tablet.cc:2369] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:58:54.871469 22224 raft_consensus.cc:482] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:58:54.871920 22224 raft_consensus.cc:504] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.873860 22224 leader_election.cc:288] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b53d55455ee48c39f304929899667f5 (127.20.169.65:45149), 99c5ab644e494a2d84b91f64a1e3e730 (127.20.169.66:41811)
I20260927 04:58:54.879770 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.874196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:41811 (local address 127.20.169.67:50995)
0927 04:58:54.874485 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:54.874528 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:54.874692 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:54.875192 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:54.875197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:54.875221 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:54.875485 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.875494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.876474 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.876482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.877891 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.877906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.879002 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.879008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.879212 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.879243 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 04:58:54.879351 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:58:54.879493 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"spinlock_wait_cycles":19712}
I20260927 04:58:54.879884 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.874719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:50995 (local address 127.20.169.66:41811)
0927 04:58:54.874923 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.874927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.874941 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:58:54.875001 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.875004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.875059 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.875180 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:54.875637 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.876346 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.878043 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.878804 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.879031 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.879108 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.879398 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.879531 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 04:58:54.881261 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.874342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45149 (local address 127.20.169.67:53307)
0927 04:58:54.875129 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:54.875143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:54.875320 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:54.875627 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:54.875632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:54.875652 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:54.875951 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.875960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.877000 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.877005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.878702 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:54.878712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.880426 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.880434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:54.880651 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.880690 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 04:58:54.880802 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:58:54.880945 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":2,"thread_start_us":491,"threads_started":1}
I20260927 04:58:54.881488 22018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20edaaf4b9e24ead9001ac1d1be99f40" candidate_uuid: "5343fe4ecc094a80a4d03b6079fe6008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c5ab644e494a2d84b91f64a1e3e730" is_pre_election: true
I20260927 04:58:54.881820 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.874355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:53307 (local address 127.20.169.65:45149)
0927 04:58:54.874563 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.874580 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.875239 (+ 659us) server_negotiation.cc:407] Connection header received
0927 04:58:54.875463 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.875466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.875519 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.875614 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:54.876099 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.876800 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.878884 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.880173 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.880372 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.880475 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.881496 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.881600 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 04:58:54.882292 22018 raft_consensus.cc:2454] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5343fe4ecc094a80a4d03b6079fe6008 in term 0.
I20260927 04:58:54.883385 22086 leader_election.cc:302] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5343fe4ecc094a80a4d03b6079fe6008, 99c5ab644e494a2d84b91f64a1e3e730; no voters:
I20260927 04:58:54.883224 21883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20edaaf4b9e24ead9001ac1d1be99f40" candidate_uuid: "5343fe4ecc094a80a4d03b6079fe6008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b53d55455ee48c39f304929899667f5" is_pre_election: true
I20260927 04:58:54.883947 22224 raft_consensus.cc:2781] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:58:54.883922 21883 raft_consensus.cc:2454] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5343fe4ecc094a80a4d03b6079fe6008 in term 0.
I20260927 04:58:54.884155 22224 raft_consensus.cc:482] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:58:54.884338 22224 raft_consensus.cc:3037] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:54.887581 22224 raft_consensus.cc:504] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.888509 22224 leader_election.cc:288] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [CANDIDATE]: Term 1 election: Requested vote from peers 5b53d55455ee48c39f304929899667f5 (127.20.169.65:45149), 99c5ab644e494a2d84b91f64a1e3e730 (127.20.169.66:41811)
I20260927 04:58:54.889016 21883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20edaaf4b9e24ead9001ac1d1be99f40" candidate_uuid: "5343fe4ecc094a80a4d03b6079fe6008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b53d55455ee48c39f304929899667f5"
I20260927 04:58:54.889107 22018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20edaaf4b9e24ead9001ac1d1be99f40" candidate_uuid: "5343fe4ecc094a80a4d03b6079fe6008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c5ab644e494a2d84b91f64a1e3e730"
I20260927 04:58:54.889308 21883 raft_consensus.cc:3037] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:54.889408 22018 raft_consensus.cc:3037] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:54.892493 22018 raft_consensus.cc:2454] T 20edaaf4b9e24ead9001ac1d1be99f40 P 99c5ab644e494a2d84b91f64a1e3e730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5343fe4ecc094a80a4d03b6079fe6008 in term 1.
I20260927 04:58:54.892513 21883 raft_consensus.cc:2454] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5b53d55455ee48c39f304929899667f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5343fe4ecc094a80a4d03b6079fe6008 in term 1.
I20260927 04:58:54.892999 22086 leader_election.cc:302] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5343fe4ecc094a80a4d03b6079fe6008, 5b53d55455ee48c39f304929899667f5, 99c5ab644e494a2d84b91f64a1e3e730; no voters:
I20260927 04:58:54.893503 22224 raft_consensus.cc:2781] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:54.894269 22224 raft_consensus.cc:686] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [term 1 LEADER]: Becoming Leader. State: Replica: 5343fe4ecc094a80a4d03b6079fe6008, State: Running, Role: LEADER
I20260927 04:58:54.894891 22224 consensus_queue.cc:237] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } }
I20260927 04:58:54.901273 21738 catalog_manager.cc:5495] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5343fe4ecc094a80a4d03b6079fe6008 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "5343fe4ecc094a80a4d03b6079fe6008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b53d55455ee48c39f304929899667f5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99c5ab644e494a2d84b91f64a1e3e730" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 41811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5343fe4ecc094a80a4d03b6079fe6008" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45129 } health_report { overall_health: HEALTHY } } }
Sep 27 04:58:54 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:58:54.962291 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:54.941121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.20.169.65:45149)
0927 04:58:54.941361 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:54.941371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:54.942739 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 04:58:54.942982 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:54.942987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:54.943065 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:54.943219 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:54.945735 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.946737 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.948395 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:54.949055 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:54.949191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:54.954893 (+ 5702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:54.954909 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:54.954926 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:54.954962 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:54.958903 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:54.959764 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:54.959771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:54.959773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:54.959982 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:54.960561 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:54.960565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:54.960568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:54.961077 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:54.961347 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:54.961642 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 04:58:54.961852 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 04:58:54 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:58:55.022531 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.002312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.20.169.65:45149)
0927 04:58:55.002631 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:55.002641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:55.003167 (+ 526us) server_negotiation.cc:407] Connection header received
0927 04:58:55.003318 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:55.003321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:55.003427 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:55.003514 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:55.006139 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.007305 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.009103 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.009688 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.009801 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:55.015188 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:55.015194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:55.015197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:55.015237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:55.019298 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.020074 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.020078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.020080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.020260 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.021008 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.021012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.021014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.021405 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:55.021650 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:55.021909 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 04:58:55.022093 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:58:55.073355 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.054836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.20.169.65:45149)
0927 04:58:55.055085 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:55.055095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:55.055560 (+ 465us) server_negotiation.cc:407] Connection header received
0927 04:58:55.055711 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:55.055715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:55.055780 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:55.055868 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:55.058188 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.059033 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.060770 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.061481 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.061579 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:55.066976 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:55.066982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:55.066985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:55.067020 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:55.070383 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.071145 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.071149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.071151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.071310 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.071660 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.071663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.071665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.072044 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:55.072253 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:55.072600 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 04:58:55.072862 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 04:58:55.075373 21843 tablet_service.cc:2908] Rejecting scan request for tablet 20edaaf4b9e24ead9001ac1d1be99f40: Uninitialized: clean time has not yet been initialized
I20260927 04:58:55.078838 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21800
I20260927 04:58:55.093880 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.079841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.20.169.65:45149)
0927 04:58:55.080063 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:55.080073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:55.080510 (+ 437us) server_negotiation.cc:407] Connection header received
0927 04:58:55.080697 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:55.080701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:55.080770 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:55.080942 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:55.082039 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.082761 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.084159 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.084611 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.084744 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:55.087559 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:55.087564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:55.087567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:55.087594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:55.090831 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.091581 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.091586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.091596 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.091732 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.092155 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.092160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.092170 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.092487 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:55.092864 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:55.093203 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 04:58:55.093508 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 04:58:55.094599 21924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:55.186363 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21800
I20260927 04:58:55.204736 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21935
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:58:55.220990 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.205678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.20.169.66:41811)
0927 04:58:55.205932 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:55.205937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:55.205960 (+ 23us) server_negotiation.cc:407] Connection header received
0927 04:58:55.206091 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:55.206096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:55.206170 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:55.206282 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:55.207927 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.208701 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.210169 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.210597 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.210737 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:55.215254 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:55.215278 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:55.215294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:55.215332 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:55.218405 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.219085 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.219092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.219094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.219214 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.219580 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.219583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.219585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.219961 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:55.220234 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:55.220524 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 04:58:55.220682 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 04:58:55.221665 22059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:55.283004 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.282491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45149 (local address 127.20.169.67:59601)
0927 04:58:55.282704 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:55.282854 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 04:58:55.283295 22085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45149: connect: Connection refused (error 111)
I20260927 04:58:55.284727 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.284375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45149 (local address 127.20.169.67:57601)
0927 04:58:55.284515 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:55.284604 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 04:58:55.284940 22085 consensus_peers.cc:487] T 20edaaf4b9e24ead9001ac1d1be99f40 P 5343fe4ecc094a80a4d03b6079fe6008 -> Peer 5b53d55455ee48c39f304929899667f5 (127.20.169.65:45149): Couldn't send request to peer 5b53d55455ee48c39f304929899667f5. Status: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:58:55.308977 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21935
I20260927 04:58:55.327167 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22069
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:58:55.342607 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:55.327948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.20.169.67:45129)
0927 04:58:55.328486 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:55.328491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:55.328510 (+ 19us) server_negotiation.cc:407] Connection header received
0927 04:58:55.328563 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:55.328566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:55.328626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:55.328748 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:55.330078 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.330770 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.332079 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:55.332493 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:55.332634 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:55.336686 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:55.336700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:55.336717 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:55.336749 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:55.339718 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.340438 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.340446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.340448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.340587 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:55.341058 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:55.341069 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:55.341072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:55.341407 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:55.341709 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:55.342008 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 04:58:55.342239 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260927 04:58:55.343389 22192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:55.433712 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 22069
I20260927 04:58:55.452560 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 21706
I20260927 04:58:55.453823 21769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:58:55.561566 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 21706
2026-09-27T04:58:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4400 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[22248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[22248](info): set up 2 sockets
Sep 27 04:58:55 dist-test-slave-39f2 krb5kdc[22248](info): commencing operation
krb5kdc: starting...
W20260927 04:58:56.756471 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:58:56 dist-test-slave-39f2 krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:58:56.953158 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:43165
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:43165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:57.264494 22261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:57.264776 22261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:57.264937 22261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:57.272011 22261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:58:57.272099 22261 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 04:58:57.272186 22261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:57.272329 22261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:57.272403 22261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:58:57.272493 22261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:58:57.281961 22261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:43165
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:57.283155 22261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:57.284452 22261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:57.297142 22269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:57.297127 22266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:57.297104 22267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:57.297823 22261 server_base.cc:1034] running on GCE node
I20260927 04:58:57.298682 22261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:57.300186 22261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:57.301456 22261 hybrid_clock.cc:648] HybridClock initialized: now 1790485137301404 us; error 57 us; skew 500 ppm
I20260927 04:58:57.301916 22261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:57 dist-test-slave-39f2 krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:57.305344 22261 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:58:57.306907 22261 webserver.cc:460] Webserver started at http://127.20.169.126:36941/ using document root <none> and password file <none>
I20260927 04:58:57.307678 22261 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:57.307821 22261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:57.308169 22261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:57.311363 22261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "1dc70c2e95a34938a092c16e647e922b"
format_stamp: "Formatted at 2026-09-27 04:58:57 on dist-test-slave-39f2"
server_key: "9d3577dc2707212e923a71e0d80f5ca5"
server_key_iv: "82e24fe8ec4d7313527b2078bc22c328"
server_key_version: "encryptionkey@0"
I20260927 04:58:57.312209 22261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "1dc70c2e95a34938a092c16e647e922b"
format_stamp: "Formatted at 2026-09-27 04:58:57 on dist-test-slave-39f2"
server_key: "9d3577dc2707212e923a71e0d80f5ca5"
server_key_iv: "82e24fe8ec4d7313527b2078bc22c328"
server_key_version: "encryptionkey@0"
I20260927 04:58:57.317764 22261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 04:58:57.321452 22276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:57.322793 22261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 04:58:57.323011 22261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "1dc70c2e95a34938a092c16e647e922b"
format_stamp: "Formatted at 2026-09-27 04:58:57 on dist-test-slave-39f2"
server_key: "9d3577dc2707212e923a71e0d80f5ca5"
server_key_iv: "82e24fe8ec4d7313527b2078bc22c328"
server_key_version: "encryptionkey@0"
I20260927 04:58:57.323230 22261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:57.348663 22261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:57.353123 22261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:57.353410 22261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:57.374862 22261 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:43165
I20260927 04:58:57.374884 22337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:43165 every 8 connection(s)
I20260927 04:58:57.376184 22261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:58:57.380193 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:57.380472 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 22261
I20260927 04:58:57.380734 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:58:57.381757 21157 external_mini_cluster.cc:1428] Setting key b71f5df60d2d0b04b8105bcaf225768f
I20260927 04:58:57.395154 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: Bootstrap starting.
Sep 27 04:58:57 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:57.398929 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:57.400022 22338 log.cc:824] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:57.403285 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: No bootstrap required, opened a new log
I20260927 04:58:57.407150 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:57.383944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.20.169.126:43165)
0927 04:58:57.384749 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:57.384761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:57.384800 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:58:57.386961 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:57.386984 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:57.387344 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:57.387655 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:57.389500 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.390859 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.392923 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.393628 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.393770 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:57.398775 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:57.398783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:57.398822 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:57.398865 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:57.402757 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:57.403489 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:57.403497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:57.403513 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:57.403690 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:57.404112 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:57.404118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:57.404121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:57.404741 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:57.405184 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:57.405695 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 04:58:57.406132 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":383,"threads_started":1}
I20260927 04:58:57.412438 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc70c2e95a34938a092c16e647e922b" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43165 } }
I20260927 04:58:57.412887 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:57.413015 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc70c2e95a34938a092c16e647e922b, State: Initialized, Role: FOLLOWER
I20260927 04:58:57.413717 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc70c2e95a34938a092c16e647e922b" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43165 } }
I20260927 04:58:57.414048 22338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:58:57.414227 22338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:58:57.414423 22338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:57.417676 22338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc70c2e95a34938a092c16e647e922b" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43165 } }
I20260927 04:58:57.418203 22338 leader_election.cc:302] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dc70c2e95a34938a092c16e647e922b; no voters:
I20260927 04:58:57.418777 22338 leader_election.cc:288] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:58:57.418891 22343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:57.419641 22343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [term 1 LEADER]: Becoming Leader. State: Replica: 1dc70c2e95a34938a092c16e647e922b, State: Running, Role: LEADER
I20260927 04:58:57.420220 22343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc70c2e95a34938a092c16e647e922b" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43165 } }
I20260927 04:58:57.421061 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:58:57.422982 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dc70c2e95a34938a092c16e647e922b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc70c2e95a34938a092c16e647e922b" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43165 } } }
I20260927 04:58:57.423360 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [sys.catalog]: This master's current role is: LEADER
I20260927 04:58:57.422958 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dc70c2e95a34938a092c16e647e922b. Latest consensus state: current_term: 1 leader_uuid: "1dc70c2e95a34938a092c16e647e922b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc70c2e95a34938a092c16e647e922b" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43165 } } }
I20260927 04:58:57.424963 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b [sys.catalog]: This master's current role is: LEADER
I20260927 04:58:57.425539 22348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:58:57.430063 22348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:58:57.440269 22348 catalog_manager.cc:1347] Generated new cluster ID: abd38b23eaa54b588bc558773dd1b423
I20260927 04:58:57.440377 22348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:58:57.449034 22348 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:58:57.450466 22348 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:58:57.460631 22348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: Generated new TSK 0
I20260927 04:58:57.461613 22348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:58:57.514523 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43165
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:57.849344 22364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:57.849606 22364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:57.849835 22364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:57.857151 22364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:57.857251 22364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:57.857441 22364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:58:57.857583 22364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:58:57.868474 22364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:57.870303 22364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:57.872229 22364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:57.886804 22369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:57.886849 22370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:57.886853 22372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:57.890211 22364 server_base.cc:1034] running on GCE node
I20260927 04:58:57.890769 22364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:57.891821 22364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:57.893105 22364 hybrid_clock.cc:648] HybridClock initialized: now 1790485137893021 us; error 49 us; skew 500 ppm
I20260927 04:58:57.893450 22364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:57 dist-test-slave-39f2 krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:57.900913 22364 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:58:57.902496 22364 webserver.cc:460] Webserver started at http://127.20.169.65:38749/ using document root <none> and password file <none>
I20260927 04:58:57.903306 22364 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:57.903442 22364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:57.903762 22364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:57.906814 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "eb02f270efd54335a6a880e9a1466587"
format_stamp: "Formatted at 2026-09-27 04:58:57 on dist-test-slave-39f2"
server_key: "e7e62e9ef7d7d25a9a093ea558169455"
server_key_iv: "5b0a7000aecc01e8be41d7886471d873"
server_key_version: "encryptionkey@0"
I20260927 04:58:57.907660 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "eb02f270efd54335a6a880e9a1466587"
format_stamp: "Formatted at 2026-09-27 04:58:57 on dist-test-slave-39f2"
server_key: "e7e62e9ef7d7d25a9a093ea558169455"
server_key_iv: "5b0a7000aecc01e8be41d7886471d873"
server_key_version: "encryptionkey@0"
I20260927 04:58:57.912787 22364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 04:58:57.916319 22379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:57.917711 22364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 04:58:57.917908 22364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "eb02f270efd54335a6a880e9a1466587"
format_stamp: "Formatted at 2026-09-27 04:58:57 on dist-test-slave-39f2"
server_key: "e7e62e9ef7d7d25a9a093ea558169455"
server_key_iv: "5b0a7000aecc01e8be41d7886471d873"
server_key_version: "encryptionkey@0"
I20260927 04:58:57.918130 22364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:57.931881 22364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:57.936146 22364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:57.936424 22364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:57.939116 22364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:57.939224 22364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:57.939385 22364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:57.939492 22364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:58:57 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:57.961472 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:57.939579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:46775 (local address 127.20.169.126:43165)
0927 04:58:57.940369 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:57.940375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:57.941506 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 04:58:57.944799 (+ 3293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:57.944841 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:57.944936 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:57.945136 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:57.946598 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.947325 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.948983 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.949603 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.949740 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:57.955206 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:57.955211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:57.955214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:57.955278 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:57.958406 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:57.959126 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:57.959132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:57.959135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:57.959267 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:57.959715 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:57.959720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:57.959729 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:57.960006 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:57.960161 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:57.960891 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 04:58:57.961108 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":554,"threads_started":1}
I20260927 04:58:57.962436 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:57.940330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.20.169.65:46775)
0927 04:58:57.941283 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:57.941360 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:57.944330 (+ 2970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:57.945366 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:57.945374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:57.945874 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:57.946431 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:57.946447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.947483 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.947488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:57.948785 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:57.948797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.949847 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.949855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:57.950089 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:57.950897 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:57.950933 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:57.954984 (+ 4051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:57.958553 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:57.958564 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:57.958582 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:57.958998 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:57.959383 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:57.959388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:57.959392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:57.959583 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:57.960137 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:57.960145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:57.960491 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 04:58:57.960744 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 04:58:57.961285 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":403,"threads_started":1}
I20260927 04:58:57.982777 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:57.968481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:58763 (local address 127.20.169.126:43165)
0927 04:58:57.968662 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:57.968669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:57.969452 (+ 783us) server_negotiation.cc:407] Connection header received
0927 04:58:57.969648 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:57.969653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:57.969721 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:57.969818 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:57.970693 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.971391 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.972691 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.973352 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.973487 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:57.976491 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:57.976499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:57.976506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:57.976548 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:57.979804 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:57.980725 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:57.980733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:57.980737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:57.980934 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:57.981308 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:57.981315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:57.981321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:57.981648 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:57.981838 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:57.982390 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 04:58:57.982556 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:58:57.983397 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:57.968565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.20.169.65:58763)
0927 04:58:57.969340 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:57.969362 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:57.969523 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:57.969840 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:57.969844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:57.970271 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:57.970555 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:57.970563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.971544 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.971551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:57.972549 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:57.972560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:57.973709 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:57.973715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:57.974039 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:57.974810 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:57.974839 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:57.976284 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:57.979943 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:57.979948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:57.979952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:57.980588 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:57.981046 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:57.981051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:57.981055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:57.981196 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:57.981829 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:57.981837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:57.982099 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 04:58:57.982309 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 04:58:57.982741 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":406,"threads_started":1}
I20260927 04:58:57.999915 22364 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:39581
I20260927 04:58:57.999944 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:39581 every 8 connection(s)
I20260927 04:58:58.001269 22364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:58:58.003886 22500 heartbeater.cc:347] Connected to a master server at 127.20.169.126:43165
I20260927 04:58:58.003895 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 22364
I20260927 04:58:58.004132 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:58:58.004139 22500 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:58.004770 21157 external_mini_cluster.cc:1428] Setting key cdcc04b4ddfdf870b023148f723cbe7f
I20260927 04:58:58.004949 22500 heartbeater.cc:511] Master 127.20.169.126:43165 requested a full tablet report, sending...
I20260927 04:58:58.007344 22292 ts_manager.cc:194] Registered new tserver with Master: eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581)
I20260927 04:58:58.009656 22292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:46775
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:58:58.046278 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43165
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:58.370846 22505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:58.371106 22505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:58.371199 22505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:58.378409 22505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:58.378508 22505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:58.378651 22505 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:58:58.378764 22505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:58:58.388240 22505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:58.389482 22505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:58.390738 22505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:58.402981 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:58.403106 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:58.404137 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:58.404949 22505 server_base.cc:1034] running on GCE node
I20260927 04:58:58.405469 22505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:58.406263 22505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:58.407511 22505 hybrid_clock.cc:648] HybridClock initialized: now 1790485138407451 us; error 38 us; skew 500 ppm
I20260927 04:58:58.407796 22505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:58 dist-test-slave-39f2 krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485138, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:58.410938 22505 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:58:58.412446 22505 webserver.cc:460] Webserver started at http://127.20.169.66:44869/ using document root <none> and password file <none>
I20260927 04:58:58.413241 22505 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:58.413378 22505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:58.413688 22505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:58.416476 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "d9d20f78667b4d32aa477221a5daefd7"
format_stamp: "Formatted at 2026-09-27 04:58:58 on dist-test-slave-39f2"
server_key: "b2a30b9e8134b30a8795c80006412151"
server_key_iv: "5aca56f7dcba3f5592ea1e00a981b3e9"
server_key_version: "encryptionkey@0"
I20260927 04:58:58.417413 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d20f78667b4d32aa477221a5daefd7"
format_stamp: "Formatted at 2026-09-27 04:58:58 on dist-test-slave-39f2"
server_key: "b2a30b9e8134b30a8795c80006412151"
server_key_iv: "5aca56f7dcba3f5592ea1e00a981b3e9"
server_key_version: "encryptionkey@0"
I20260927 04:58:58.423261 22505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:58:58.426756 22520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:58.428093 22505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 04:58:58.428270 22505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "d9d20f78667b4d32aa477221a5daefd7"
format_stamp: "Formatted at 2026-09-27 04:58:58 on dist-test-slave-39f2"
server_key: "b2a30b9e8134b30a8795c80006412151"
server_key_iv: "5aca56f7dcba3f5592ea1e00a981b3e9"
server_key_version: "encryptionkey@0"
I20260927 04:58:58.428467 22505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:58.450726 22505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:58.454883 22505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:58.455144 22505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:58.457969 22505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:58.458050 22505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:58.458215 22505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:58.458272 22505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:58:58 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485138, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:58.483183 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.458747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:50071 (local address 127.20.169.126:43165)
0927 04:58:58.458954 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:58.458960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:58.460443 (+ 1483us) server_negotiation.cc:407] Connection header received
0927 04:58:58.462567 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:58.462573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:58.462665 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:58.462791 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:58.465270 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.466419 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.469057 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.469770 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.469943 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.475694 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:58.475701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:58.475705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:58.475751 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:58.479196 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:58.480214 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:58.480222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:58.480228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:58.480374 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:58.481006 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:58.481011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:58.481015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:58.481462 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:58.481629 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:58.482608 (+ 979us) server_negotiation.cc:299] Negotiation successful
0927 04:58:58.482841 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 04:58:58.484721 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.459358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.20.169.66:50071)
0927 04:58:58.460240 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:58.460291 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:58.462273 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:58.463072 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:58.463087 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:58.464102 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:58.465058 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.465089 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.466573 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.466581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.468913 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.468928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.470105 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.470117 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.470544 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.471452 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:58.471491 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:58.475377 (+ 3886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:58.479386 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.479393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.479406 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:58.480074 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:58.480488 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.480498 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.480519 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:58.480866 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:58.481601 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:58.481610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:58.482146 (+ 536us) client_negotiation.cc:769] Sending connection context
0927 04:58:58.482523 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 04:58:58.483120 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":395,"threads_started":1}
I20260927 04:58:58.503684 22505 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45193
I20260927 04:58:58.503770 22641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45193 every 8 connection(s)
I20260927 04:58:58.505445 22505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:58:58.507711 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.492058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.20.169.66:41903)
0927 04:58:58.493021 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:58.493037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:58.493142 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:58.493507 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:58.493510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:58.493961 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:58.494228 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.494236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.496030 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.496034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.497150 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.497165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.498402 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.498410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.498634 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.499446 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:58.499483 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:58.500900 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:58.504445 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.504450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.504454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:58.505028 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:58.505530 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.505555 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.505561 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:58.505706 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:58.506568 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:58.506579 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:58.506827 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 04:58:58.506960 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 04:58:58.507253 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":573,"threads_started":1}
I20260927 04:58:58.507758 22642 heartbeater.cc:347] Connected to a master server at 127.20.169.126:43165
I20260927 04:58:58.508004 22642 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:58.508533 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.492583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:41903 (local address 127.20.169.126:43165)
0927 04:58:58.492748 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:58.492753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:58.493126 (+ 373us) server_negotiation.cc:407] Connection header received
0927 04:58:58.493239 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:58.493248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:58.493360 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:58.493495 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:58.494375 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.495885 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.497365 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.498131 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.498276 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.501117 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:58.501125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:58.501131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:58.501170 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:58.504319 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:58.505201 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:58.505205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:58.505208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:58.505389 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:58.505847 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:58.505853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:58.505859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:58.506420 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:58.507976 (+ 1556us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:58.508121 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 04:58:58.508323 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:58:58.508720 22642 heartbeater.cc:511] Master 127.20.169.126:43165 requested a full tablet report, sending...
I20260927 04:58:58.510272 22292 ts_manager.cc:194] Registered new tserver with Master: d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66:45193)
I20260927 04:58:58.511330 22292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:50071
I20260927 04:58:58.514564 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 22505
I20260927 04:58:58.514765 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:58:58.515219 21157 external_mini_cluster.cc:1428] Setting key 988921b4ab1e9920adbfe22a2c6b0b7b
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:58:58.556758 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43165
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:58:58.874303 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:58:58.874607 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:58:58.874776 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:58:58.881860 22646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:58:58.882004 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:58:58.882170 22646 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:58:58.882306 22646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:58:58.891901 22646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:58:58.893167 22646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:58:58.894460 22646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:58:58.907963 22651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:58.907939 22652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:58:58.908195 22654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:58:58.908785 22646 server_base.cc:1034] running on GCE node
I20260927 04:58:58.909543 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:58:58.910530 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:58:58.911814 22646 hybrid_clock.cc:648] HybridClock initialized: now 1790485138911703 us; error 76 us; skew 500 ppm
I20260927 04:58:58.912168 22646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:58:58 dist-test-slave-39f2 krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485138, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:58:58.915799 22646 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:58:58.917408 22646 webserver.cc:460] Webserver started at http://127.20.169.67:41077/ using document root <none> and password file <none>
I20260927 04:58:58.918136 22646 fs_manager.cc:359] Metadata directory not provided
I20260927 04:58:58.918269 22646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:58:58.918607 22646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:58:58.921917 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "7c3f19266f53435e8cb2eeda3684c4f5"
format_stamp: "Formatted at 2026-09-27 04:58:58 on dist-test-slave-39f2"
server_key: "e74536f9d7820370811e0d6ceae45d7b"
server_key_iv: "bf70b5a2b9b9ae814ba64648b7e72771"
server_key_version: "encryptionkey@0"
I20260927 04:58:58.922910 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "7c3f19266f53435e8cb2eeda3684c4f5"
format_stamp: "Formatted at 2026-09-27 04:58:58 on dist-test-slave-39f2"
server_key: "e74536f9d7820370811e0d6ceae45d7b"
server_key_iv: "bf70b5a2b9b9ae814ba64648b7e72771"
server_key_version: "encryptionkey@0"
I20260927 04:58:58.928484 22646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 04:58:58.932335 22661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:58.933736 22646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 04:58:58.933920 22646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "7c3f19266f53435e8cb2eeda3684c4f5"
format_stamp: "Formatted at 2026-09-27 04:58:58 on dist-test-slave-39f2"
server_key: "e74536f9d7820370811e0d6ceae45d7b"
server_key_iv: "bf70b5a2b9b9ae814ba64648b7e72771"
server_key_version: "encryptionkey@0"
I20260927 04:58:58.934145 22646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:58:58.951015 22646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:58:58.955768 22646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:58:58.956067 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:58:58.958866 22646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:58:58.958998 22646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:58:58.959162 22646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:58:58.959236 22646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:58:58 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485138, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:58.981652 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.959246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56163 (local address 127.20.169.126:43165)
0927 04:58:58.959452 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:58.959459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:58.960741 (+ 1282us) server_negotiation.cc:407] Connection header received
0927 04:58:58.962769 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:58.962775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:58.962864 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:58.962975 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:58.965145 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.966642 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.969145 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.969692 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.969820 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.974492 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:58.974497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:58.974500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:58.974532 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:58.977922 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:58.978871 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:58.978885 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:58.978888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:58.979032 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:58.979482 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:58.979487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:58.979490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:58.979857 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:58.980016 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:58.981064 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 04:58:58.981291 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 04:58:58.982585 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.959715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.20.169.67:56163)
0927 04:58:58.960553 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:58.960609 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:58.962456 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:58.963213 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:58.963221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:58.963938 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:58.964923 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.964957 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.966826 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.966832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.968995 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.969006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.969930 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.969938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.970175 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.970936 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:58.970957 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:58.974181 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:58.978056 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.978064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.978080 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:58.978742 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:58.979146 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.979155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.979158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:58.979348 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:58.980088 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:58.980093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:58.980546 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 04:58:58.980922 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 04:58:58.981430 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":351,"threads_started":1}
I20260927 04:58:59.002086 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.988494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:55191 (local address 127.20.169.126:43165)
0927 04:58:58.988646 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:58.988650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:58.989470 (+ 820us) server_negotiation.cc:407] Connection header received
0927 04:58:58.989656 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:58.989661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:58.989727 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:58.989823 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:58.991242 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.992174 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.994183 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.994638 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.994783 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.996387 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:58.996393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:58.996398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:58.996435 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:58.999483 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:59.000133 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:59.000139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:59.000142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:59.000266 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:59.000627 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:59.000632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:59.000635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:59.000981 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:59.001132 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.001515 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.001724 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:58:59.002158 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:58.988595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.20.169.67:55191)
0927 04:58:58.989365 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:58.989395 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:58.989534 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:58.989845 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:58.989849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:58.990564 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:58.991059 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.991079 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.992366 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.992383 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.994018 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:58.994038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:58.994823 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:58.994829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:58.994999 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:58.995560 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:58.995575 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:58.996202 (+ 627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:58.999641 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:58.999646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:58.999650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:59.000011 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:59.000382 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:59.000387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:59.000391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:59.000511 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:59.001096 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:59.001101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:59.001274 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.001421 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.001725 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":414,"threads_started":1}
I20260927 04:58:59.012671 22500 heartbeater.cc:503] Master 127.20.169.126:43165 was elected leader, sending a full tablet report...
I20260927 04:58:59.023412 22646 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:39075
I20260927 04:58:59.023444 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:39075 every 8 connection(s)
I20260927 04:58:59.024601 22646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:58:59.026887 22782 heartbeater.cc:347] Connected to a master server at 127.20.169.126:43165
I20260927 04:58:59.027115 22782 heartbeater.cc:464] Registering TS with master...
I20260927 04:58:59.027870 22782 heartbeater.cc:511] Master 127.20.169.126:43165 requested a full tablet report, sending...
I20260927 04:58:59.029403 22292 ts_manager.cc:194] Registered new tserver with Master: 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075)
I20260927 04:58:59.030275 22292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:56163
I20260927 04:58:59.034943 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 22646
I20260927 04:58:59.035092 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:58:59.035475 21157 external_mini_cluster.cc:1428] Setting key cd6f1cd3fda8295aab342746c0ce7751
I20260927 04:58:59.038473 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:58:59 dist-test-slave-39f2 krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:58:59 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:59.086127 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.070990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.20.169.126:43165)
0927 04:58:59.071236 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.071243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.071430 (+ 187us) server_negotiation.cc:407] Connection header received
0927 04:58:59.071584 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.071588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.071657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.071745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:59.073190 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.074247 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.075594 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.076036 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.076140 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.080025 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:59.080031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:59.080035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:59.080073 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:59.083247 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:59.083946 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:59.083951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:59.083953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:59.084081 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:59.084530 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:59.084533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:59.084535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:59.084918 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:59.085126 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.085519 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.085705 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 04:58:59.090673 22292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:58:59.098155 22292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:58:59.124651 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.110751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:39075 (local address 127.0.0.1:53508)
0927 04:58:59.111882 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.111926 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.112113 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.113790 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.113803 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.113848 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.114445 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.114462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.117724 (+ 3262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.117734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.121068 (+ 3334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.121089 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.123411 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.123421 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.123738 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.123802 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.123919 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.124133 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":3,"thread_start_us":725,"threads_started":1}
I20260927 04:58:59.125422 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.110994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.20.169.67:39075)
0927 04:58:59.112927 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.112940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.112985 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:58:59.113116 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.113128 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.113475 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.114738 (+ 1263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.115176 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.117494 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.121405 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.123161 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.124454 (+ 1293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.124716 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.124851 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.125023 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1701,"thread_start_us":1494,"threads_started":1}
I20260927 04:58:59.129678 22712 tablet_service.cc:1467] Processing CreateTablet for tablet bd3aa4e464734e30a64128537897c03b (DEFAULT_TABLE table=test-table [id=595c1e30b69e48feae6a5f414a68f3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:59.131242 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3aa4e464734e30a64128537897c03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.132334 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.111787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.20.169.66:45193)
0927 04:58:59.122590 (+ 10803us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.122601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.122634 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:58:59.122734 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.122742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.123026 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.123802 (+ 776us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.125681 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.127210 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.129527 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.130931 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.131116 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.131353 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.131843 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.132002 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10619,"thread_start_us":11373,"threads_started":1}
I20260927 04:58:59.132874 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.111126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45193 (local address 127.0.0.1:55688)
0927 04:58:59.112573 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.112592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.112709 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.123458 (+ 10749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.123462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.123491 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.124219 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.124232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.127374 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.127381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.129320 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.129338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.131238 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.131247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.131629 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.131655 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.131782 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.131926 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":285,"thread_start_us":1216,"threads_started":1}
I20260927 04:58:59.133762 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.112787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39581 (local address 127.0.0.1:33666)
0927 04:58:59.125481 (+ 12694us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.125701 (+ 220us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.125809 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.127016 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.127020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.127045 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.127236 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.127248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.129140 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.129145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.131059 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.131073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.132965 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.132970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.133200 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.133269 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.133363 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.133492 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12587,"thread_start_us":13317,"threads_started":1}
I20260927 04:58:59.134756 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.113168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.20.169.65:39581)
0927 04:58:59.126318 (+ 13150us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.126328 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.126358 (+ 30us) server_negotiation.cc:407] Connection header received
0927 04:58:59.126518 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.126526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.126768 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.127007 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.127421 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.128962 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.131266 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.132740 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.134014 (+ 1274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.134223 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.134287 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.134457 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12974,"thread_start_us":13999,"threads_started":1}
I20260927 04:58:59.137110 22571 tablet_service.cc:1467] Processing CreateTablet for tablet bd3aa4e464734e30a64128537897c03b (DEFAULT_TABLE table=test-table [id=595c1e30b69e48feae6a5f414a68f3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:59.138629 22571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3aa4e464734e30a64128537897c03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.138357 22435 tablet_service.cc:1467] Processing CreateTablet for tablet bd3aa4e464734e30a64128537897c03b (DEFAULT_TABLE table=test-table [id=595c1e30b69e48feae6a5f414a68f3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:59.139621 22435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3aa4e464734e30a64128537897c03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.146922 22801 tablet_bootstrap.cc:492] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Bootstrap starting.
I20260927 04:58:59.151507 22801 tablet_bootstrap.cc:654] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.151770 22802 tablet_bootstrap.cc:492] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Bootstrap starting.
I20260927 04:58:59.153065 22801 log.cc:824] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:59.153512 22803 tablet_bootstrap.cc:492] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Bootstrap starting.
I20260927 04:58:59.155627 22802 tablet_bootstrap.cc:654] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.156612 22802 log.cc:824] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:59.158012 22803 tablet_bootstrap.cc:654] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.159164 22803 log.cc:824] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Log is configured to *not* fsync() on all Append() calls
I20260927 04:58:59.159452 22802 tablet_bootstrap.cc:492] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: No bootstrap required, opened a new log
I20260927 04:58:59.159453 22801 tablet_bootstrap.cc:492] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: No bootstrap required, opened a new log
I20260927 04:58:59.159724 22801 ts_tablet_manager.cc:1397] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260927 04:58:59.159724 22802 ts_tablet_manager.cc:1397] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 04:58:59.161854 22803 tablet_bootstrap.cc:492] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: No bootstrap required, opened a new log
I20260927 04:58:59.162145 22803 ts_tablet_manager.cc:1397] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20260927 04:58:59.170938 22803 raft_consensus.cc:348] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.171089 22802 raft_consensus.cc:348] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.171473 22803 raft_consensus.cc:374] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.171630 22803 raft_consensus.cc:729] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb02f270efd54335a6a880e9a1466587, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.171670 22802 raft_consensus.cc:374] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.171834 22802 raft_consensus.cc:729] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d20f78667b4d32aa477221a5daefd7, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.171631 22801 raft_consensus.cc:348] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.172215 22801 raft_consensus.cc:374] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.172413 22801 raft_consensus.cc:729] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3f19266f53435e8cb2eeda3684c4f5, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.172540 22803 consensus_queue.cc:260] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.173012 22802 consensus_queue.cc:260] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.173348 22801 consensus_queue.cc:260] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.173545 22807 raft_consensus.cc:482] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:58:59.174037 22807 raft_consensus.cc:504] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.178115 22807 leader_election.cc:288] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075), eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581)
I20260927 04:58:59.179171 22642 heartbeater.cc:503] Master 127.20.169.126:43165 was elected leader, sending a full tablet report...
I20260927 04:58:59.188895 22803 ts_tablet_manager.cc:1428] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Time spent starting tablet: real 0.027s user 0.010s sys 0.008s
I20260927 04:58:59.192158 22782 heartbeater.cc:503] Master 127.20.169.126:43165 was elected leader, sending a full tablet report...
I20260927 04:58:59.197454 22801 ts_tablet_manager.cc:1428] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Time spent starting tablet: real 0.038s user 0.021s sys 0.010s
I20260927 04:58:59.202587 22802 ts_tablet_manager.cc:1428] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Time spent starting tablet: real 0.043s user 0.025s sys 0.019s
I20260927 04:58:59.206511 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.178854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:39075 (local address 127.20.169.66:43727)
0927 04:58:59.198488 (+ 19634us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.198520 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.198708 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.199867 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.199874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.199901 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.200876 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.200890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.202000 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.202006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.204290 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.204314 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.205885 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.205889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.206004 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.206028 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.206104 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.206187 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19454,"thread_start_us":5831,"threads_started":1}
I20260927 04:58:59.206535 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.180497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:43727 (local address 127.20.169.67:39075)
0927 04:58:59.181010 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.181014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.199372 (+ 18358us) server_negotiation.cc:407] Connection header received
0927 04:58:59.199481 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.199485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.199559 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.199648 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.201047 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.201820 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.204475 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.205658 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.205818 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.205892 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.206158 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.206280 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260927 04:58:59.207576 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.180697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:39171 (local address 127.20.169.65:39581)
0927 04:58:59.181173 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.181177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.199721 (+ 18544us) server_negotiation.cc:407] Connection header received
0927 04:58:59.200012 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.200024 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.200111 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.200209 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.202000 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.203062 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.205401 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.206565 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.206722 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.206818 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.207221 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.207349 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339}
I20260927 04:58:59.207432 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3aa4e464734e30a64128537897c03b" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" is_pre_election: true
I20260927 04:58:59.207661 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.179579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39581 (local address 127.20.169.66:39171)
0927 04:58:59.199615 (+ 20036us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.199641 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.199834 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.200466 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.200469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.200491 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.201821 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.201835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.203238 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.203243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.205228 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.205259 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.206957 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.206963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.207068 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.207089 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.207183 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.207303 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19902,"thread_start_us":18254,"threads_started":1}
I20260927 04:58:59.207945 22737 raft_consensus.cc:2454] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 0.
I20260927 04:58:59.208585 22522 leader_election.cc:302] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3f19266f53435e8cb2eeda3684c4f5, d9d20f78667b4d32aa477221a5daefd7; no voters:
I20260927 04:58:59.208551 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3aa4e464734e30a64128537897c03b" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02f270efd54335a6a880e9a1466587" is_pre_election: true
I20260927 04:58:59.209033 22455 raft_consensus.cc:2454] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 0.
I20260927 04:58:59.209041 22809 raft_consensus.cc:2781] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:58:59.209182 22809 raft_consensus.cc:482] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:58:59.209479 22809 raft_consensus.cc:3037] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.212699 22809 raft_consensus.cc:504] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.213744 22809 leader_election.cc:288] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 election: Requested vote from peers 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075), eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581)
I20260927 04:58:59.214073 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3aa4e464734e30a64128537897c03b" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02f270efd54335a6a880e9a1466587"
I20260927 04:58:59.214073 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3aa4e464734e30a64128537897c03b" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3f19266f53435e8cb2eeda3684c4f5"
I20260927 04:58:59.214285 22737 raft_consensus.cc:3037] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.214329 22455 raft_consensus.cc:3037] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.218778 22455 raft_consensus.cc:2454] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 1.
I20260927 04:58:59.218793 22737 raft_consensus.cc:2454] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 1.
I20260927 04:58:59.219349 22522 leader_election.cc:302] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3f19266f53435e8cb2eeda3684c4f5, d9d20f78667b4d32aa477221a5daefd7; no voters:
I20260927 04:58:59.219667 22809 raft_consensus.cc:2781] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:59.219921 22809 raft_consensus.cc:686] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 1 LEADER]: Becoming Leader. State: Replica: d9d20f78667b4d32aa477221a5daefd7, State: Running, Role: LEADER
I20260927 04:58:59.220503 22809 consensus_queue.cc:237] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } }
I20260927 04:58:59.226815 22292 catalog_manager.cc:5495] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66). New cstate: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } health_report { overall_health: UNKNOWN } } }
W20260927 04:58:59.253964 22502 tablet.cc:2369] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:58:59.257769 22643 tablet.cc:2369] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:58:59.271311 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.263951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.20.169.66:45193)
0927 04:58:59.264213 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.264221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.264364 (+ 143us) server_negotiation.cc:407] Connection header received
0927 04:58:59.264776 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.264783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.264914 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.265071 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:58:59.265549 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.266458 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.268474 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.269222 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.269387 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.270061 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:58:59.270547 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:58:59.270710 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.270853 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.270939 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 04:58:59.276611 22783 tablet.cc:2369] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:58:59.281226 22455 raft_consensus.cc:1261] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Refusing update from remote peer d9d20f78667b4d32aa477221a5daefd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:58:59.281225 22737 raft_consensus.cc:1261] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Refusing update from remote peer d9d20f78667b4d32aa477221a5daefd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:58:59.282497 22807 consensus_queue.cc:1035] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:58:59.282922 22809 consensus_queue.cc:1035] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:58:59.299769 22815 mvcc.cc:205] Tried to move back new op lower bound from 7333827130485104640 to 7333827130252898304. Current Snapshot: MvccSnapshot[applied={T|T < 7333827130485104640}]
I20260927 04:58:59.323755 22293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:58:59.324141 22293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:58:59.329041 22293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: Sending DeleteTablet for 3 replicas of tablet bd3aa4e464734e30a64128537897c03b
I20260927 04:58:59.330626 22712 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3aa4e464734e30a64128537897c03b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:58:59 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53508
I20260927 04:58:59.330932 22571 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3aa4e464734e30a64128537897c03b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:58:59 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:55688
I20260927 04:58:59.331224 22435 tablet_service.cc:1514] Processing DeleteTablet for tablet bd3aa4e464734e30a64128537897c03b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:58:59 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:33666
I20260927 04:58:59.336138 22293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:58:59.337066 22293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:58:59.340656 22826 tablet_replica.cc:330] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: stopping tablet replica
I20260927 04:58:59.341248 22826 raft_consensus.cc:2229] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:58:59.342011 22826 raft_consensus.cc:2258] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:58:59.343379 22824 tablet_replica.cc:330] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: stopping tablet replica
I20260927 04:58:59.343871 22824 raft_consensus.cc:2229] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:58:59.344594 22824 raft_consensus.cc:2258] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:58:59.344624 22826 ts_tablet_manager.cc:1905] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:58:59.347834 22711 tablet_service.cc:1467] Processing CreateTablet for tablet 142003ed2ac644929849befb44ca23a8 (DEFAULT_TABLE table=test-table [id=004f6702adb3496dbff24c3267f44772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:59.348124 22824 ts_tablet_manager.cc:1905] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:58:59.348460 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142003ed2ac644929849befb44ca23a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.349479 22825 tablet_replica.cc:330] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: stopping tablet replica
I20260927 04:58:59.349997 22825 raft_consensus.cc:2229] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:58:59.351052 22825 raft_consensus.cc:2258] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:58:59.351370 22570 tablet_service.cc:1467] Processing CreateTablet for tablet 142003ed2ac644929849befb44ca23a8 (DEFAULT_TABLE table=test-table [id=004f6702adb3496dbff24c3267f44772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:59.352212 22570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142003ed2ac644929849befb44ca23a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.354353 22826 ts_tablet_manager.cc:1918] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:58:59.354576 22826 log.cc:1196] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/bd3aa4e464734e30a64128537897c03b
I20260927 04:58:59.355059 22435 tablet_service.cc:1467] Processing CreateTablet for tablet 142003ed2ac644929849befb44ca23a8 (DEFAULT_TABLE table=test-table [id=004f6702adb3496dbff24c3267f44772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:58:59.355511 22826 ts_tablet_manager.cc:1939] T bd3aa4e464734e30a64128537897c03b P eb02f270efd54335a6a880e9a1466587: Deleting consensus metadata
I20260927 04:58:59.355691 22435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142003ed2ac644929849befb44ca23a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.356798 22801 tablet_bootstrap.cc:492] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Bootstrap starting.
I20260927 04:58:59.357914 22279 catalog_manager.cc:4841] TS eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581): tablet bd3aa4e464734e30a64128537897c03b (table test-table [id=595c1e30b69e48feae6a5f414a68f3a7]) successfully deleted
I20260927 04:58:59.358609 22801 tablet_bootstrap.cc:654] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.363243 22824 ts_tablet_manager.cc:1918] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:58:59.363485 22824 log.cc:1196] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/bd3aa4e464734e30a64128537897c03b
I20260927 04:58:59.364324 22824 ts_tablet_manager.cc:1939] T bd3aa4e464734e30a64128537897c03b P 7c3f19266f53435e8cb2eeda3684c4f5: Deleting consensus metadata
I20260927 04:58:59.366611 22278 catalog_manager.cc:4841] TS 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075): tablet bd3aa4e464734e30a64128537897c03b (table test-table [id=595c1e30b69e48feae6a5f414a68f3a7]) successfully deleted
I20260927 04:58:59.368729 22803 tablet_bootstrap.cc:492] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Bootstrap starting.
I20260927 04:58:59.369119 22802 tablet_bootstrap.cc:492] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Bootstrap starting.
I20260927 04:58:59.371620 22803 tablet_bootstrap.cc:654] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.372211 22802 tablet_bootstrap.cc:654] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.373719 22801 tablet_bootstrap.cc:492] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: No bootstrap required, opened a new log
I20260927 04:58:59.373970 22801 ts_tablet_manager.cc:1397] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260927 04:58:59.375208 22801 raft_consensus.cc:348] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.375517 22801 raft_consensus.cc:374] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.375684 22801 raft_consensus.cc:729] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3f19266f53435e8cb2eeda3684c4f5, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.376123 22801 consensus_queue.cc:260] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.376624 22825 ts_tablet_manager.cc:1905] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:58:59.377913 22801 ts_tablet_manager.cc:1428] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 04:58:59.392234 22825 ts_tablet_manager.cc:1918] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:58:59.392541 22825 log.cc:1196] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/bd3aa4e464734e30a64128537897c03b
I20260927 04:58:59.393462 22825 ts_tablet_manager.cc:1939] T bd3aa4e464734e30a64128537897c03b P d9d20f78667b4d32aa477221a5daefd7: Deleting consensus metadata
I20260927 04:58:59.395740 22279 catalog_manager.cc:4841] TS d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66:45193): tablet bd3aa4e464734e30a64128537897c03b (table test-table [id=595c1e30b69e48feae6a5f414a68f3a7]) successfully deleted
I20260927 04:58:59.397367 22802 tablet_bootstrap.cc:492] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: No bootstrap required, opened a new log
I20260927 04:58:59.397591 22802 ts_tablet_manager.cc:1397] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.013s
I20260927 04:58:59.398725 22803 tablet_bootstrap.cc:492] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: No bootstrap required, opened a new log
I20260927 04:58:59.399031 22803 ts_tablet_manager.cc:1397] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.001s
I20260927 04:58:59.398953 22802 raft_consensus.cc:348] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.399294 22802 raft_consensus.cc:374] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.399529 22802 raft_consensus.cc:729] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d20f78667b4d32aa477221a5daefd7, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.400003 22802 consensus_queue.cc:260] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.400767 22803 raft_consensus.cc:348] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.401135 22803 raft_consensus.cc:374] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.401372 22803 raft_consensus.cc:729] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb02f270efd54335a6a880e9a1466587, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.401839 22803 consensus_queue.cc:260] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.403623 22802 ts_tablet_manager.cc:1428] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 04:58:59.406384 22803 ts_tablet_manager.cc:1428] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260927 04:58:59.594218 22810 raft_consensus.cc:482] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:58:59.594622 22810 raft_consensus.cc:504] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.597311 22810 leader_election.cc:288] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581), d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66:45193)
I20260927 04:58:59.607645 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.597732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:35689 (local address 127.20.169.65:39581)
0927 04:58:59.597970 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.597978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.598305 (+ 327us) server_negotiation.cc:407] Connection header received
0927 04:58:59.598560 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.598565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.598681 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.598764 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.599622 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.601318 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.604783 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.606523 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.606738 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.606834 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.607097 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.607232 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 04:58:59.607770 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.597620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39581 (local address 127.20.169.67:35689)
0927 04:58:59.598167 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.598187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.598359 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.599079 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.599087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.599125 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.599417 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.599444 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.601527 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.601534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.604546 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.604562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.606729 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.606735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.606938 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.606963 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.607071 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.607213 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":297,"threads_started":1}
I20260927 04:58:59.608587 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142003ed2ac644929849befb44ca23a8" candidate_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02f270efd54335a6a880e9a1466587" is_pre_election: true
I20260927 04:58:59.608911 22455 raft_consensus.cc:2454] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3f19266f53435e8cb2eeda3684c4f5 in term 0.
I20260927 04:58:59.609601 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.598625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:43617 (local address 127.20.169.66:45193)
0927 04:58:59.598825 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.598841 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.600085 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 04:58:59.600369 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.600374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.600470 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.600519 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:58:59.601401 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.602402 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.605618 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.607648 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.607823 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.607917 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.609138 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.609244 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 04:58:59.609747 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.598438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45193 (local address 127.20.169.67:43617)
0927 04:58:59.599966 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.599986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.600210 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.600656 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.600659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.600677 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:58:59.601235 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.601244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.602632 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.602637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.605367 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.605389 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.608513 (+ 3124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.608520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.608898 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.608934 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.609096 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.609268 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":1270,"threads_started":1}
I20260927 04:58:59.610265 22664 leader_election.cc:302] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3f19266f53435e8cb2eeda3684c4f5, eb02f270efd54335a6a880e9a1466587; no voters:
I20260927 04:58:59.610960 22810 raft_consensus.cc:2781] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:58:59.611228 22810 raft_consensus.cc:482] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:58:59.611407 22810 raft_consensus.cc:3037] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.611189 22591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142003ed2ac644929849befb44ca23a8" candidate_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d20f78667b4d32aa477221a5daefd7" is_pre_election: true
I20260927 04:58:59.612110 22591 raft_consensus.cc:2454] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3f19266f53435e8cb2eeda3684c4f5 in term 0.
I20260927 04:58:59.615207 22810 raft_consensus.cc:504] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.616032 22810 leader_election.cc:288] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [CANDIDATE]: Term 1 election: Requested vote from peers eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581), d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66:45193)
I20260927 04:58:59.616490 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142003ed2ac644929849befb44ca23a8" candidate_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02f270efd54335a6a880e9a1466587"
I20260927 04:58:59.616707 22455 raft_consensus.cc:3037] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.616707 22591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142003ed2ac644929849befb44ca23a8" candidate_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d20f78667b4d32aa477221a5daefd7"
I20260927 04:58:59.616994 22591 raft_consensus.cc:3037] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.618806 22455 raft_consensus.cc:2454] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3f19266f53435e8cb2eeda3684c4f5 in term 1.
I20260927 04:58:59.619174 22664 leader_election.cc:302] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3f19266f53435e8cb2eeda3684c4f5, eb02f270efd54335a6a880e9a1466587; no voters:
I20260927 04:58:59.619520 22810 raft_consensus.cc:2781] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:59.620123 22810 raft_consensus.cc:686] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3f19266f53435e8cb2eeda3684c4f5, State: Running, Role: LEADER
I20260927 04:58:59.620654 22591 raft_consensus.cc:2454] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3f19266f53435e8cb2eeda3684c4f5 in term 1.
I20260927 04:58:59.620934 22810 consensus_queue.cc:237] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.625250 22293 catalog_manager.cc:5495] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } health_report { overall_health: HEALTHY } } }
Sep 27 04:58:59 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:58:59.702165 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.687962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43165 (local address 127.0.0.1:43800)
0927 04:58:59.689174 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.689201 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.689367 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.689790 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.689795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.690492 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:58:59.690706 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.690714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.691794 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.691801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.692742 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.692752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.693502 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.693507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.693738 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.694398 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:58:59.694415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:58:59.696606 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:58:59.699582 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:59.699595 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:59.699603 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:59.700004 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:59.700338 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:58:59.700344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:58:59.700347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:58:59.700476 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:58:59.701031 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:58:59.701039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:58:59.701195 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.701370 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.701696 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":25,"thread_start_us":857,"threads_started":1}
I20260927 04:58:59.702186 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.687912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.20.169.126:43165)
0927 04:58:59.689483 (+ 1571us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.689489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.689507 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:58:59.689555 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.689558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.689642 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.689769 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:58:59.690857 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.691666 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.692943 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.693332 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.693493 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.696797 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:58:59.696842 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:58:59.696848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:58:59.696884 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:58:59.699455 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:59.700111 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:59.700119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:59.700122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:59.700237 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:58:59.700572 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:58:59.700579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:58:59.700582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:58:59.700881 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:58:59.701043 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.701443 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.701806 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":204,"server-negotiator.queue_time_us":1409,"thread_start_us":1285,"threads_started":1}
I20260927 04:58:59.709024 22293 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:43800:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 04:58:59.710265 22293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:58:59.719029 22711 tablet_service.cc:1467] Processing CreateTablet for tablet a2d11be79a3042aa99ccea0aa80e2a84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce419b3b517746bf87a5189e100512ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:58:59.719614 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d11be79a3042aa99ccea0aa80e2a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.719599 22570 tablet_service.cc:1467] Processing CreateTablet for tablet a2d11be79a3042aa99ccea0aa80e2a84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce419b3b517746bf87a5189e100512ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:58:59.720331 22570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d11be79a3042aa99ccea0aa80e2a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.725957 22435 tablet_service.cc:1467] Processing CreateTablet for tablet a2d11be79a3042aa99ccea0aa80e2a84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce419b3b517746bf87a5189e100512ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:58:59.726774 22435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d11be79a3042aa99ccea0aa80e2a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:58:59.729893 22802 tablet_bootstrap.cc:492] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: Bootstrap starting.
I20260927 04:58:59.731349 22801 tablet_bootstrap.cc:492] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: Bootstrap starting.
I20260927 04:58:59.733151 22802 tablet_bootstrap.cc:654] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.734685 22801 tablet_bootstrap.cc:654] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.738411 22803 tablet_bootstrap.cc:492] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: Bootstrap starting.
I20260927 04:58:59.741724 22803 tablet_bootstrap.cc:654] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: Neither blocks nor log segments found. Creating new log.
I20260927 04:58:59.742226 22801 tablet_bootstrap.cc:492] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: No bootstrap required, opened a new log
I20260927 04:58:59.742497 22801 ts_tablet_manager.cc:1397] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260927 04:58:59.743289 22801 raft_consensus.cc:348] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.743630 22801 raft_consensus.cc:374] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.743827 22801 raft_consensus.cc:729] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3f19266f53435e8cb2eeda3684c4f5, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.744278 22801 consensus_queue.cc:260] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.744841 22810 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.745208 22810 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.746076 22803 tablet_bootstrap.cc:492] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: No bootstrap required, opened a new log
I20260927 04:58:59.746330 22803 ts_tablet_manager.cc:1397] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260927 04:58:59.747171 22802 tablet_bootstrap.cc:492] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: No bootstrap required, opened a new log
I20260927 04:58:59.747125 22803 raft_consensus.cc:348] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.747416 22802 ts_tablet_manager.cc:1397] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.009s
I20260927 04:58:59.747465 22803 raft_consensus.cc:374] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.747630 22803 raft_consensus.cc:729] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb02f270efd54335a6a880e9a1466587, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.747941 22803 consensus_queue.cc:260] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.748420 22808 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.748761 22808 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.748745 22802 raft_consensus.cc:348] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.749043 22802 raft_consensus.cc:374] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:58:59.749146 22802 raft_consensus.cc:729] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d20f78667b4d32aa477221a5daefd7, State: Initialized, Role: FOLLOWER
I20260927 04:58:59.749619 22801 ts_tablet_manager.cc:1428] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260927 04:58:59.749745 22803 ts_tablet_manager.cc:1428] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 04:58:59.750664 22802 consensus_queue.cc:260] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.751175 22809 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.751466 22809 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.752909 22802 ts_tablet_manager.cc:1428] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 04:58:59.863052 22809 raft_consensus.cc:482] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:58:59.863300 22809 raft_consensus.cc:504] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.864045 22809 leader_election.cc:288] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581), 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075)
I20260927 04:58:59.864711 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d11be79a3042aa99ccea0aa80e2a84" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" is_pre_election: true
I20260927 04:58:59.864713 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d11be79a3042aa99ccea0aa80e2a84" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02f270efd54335a6a880e9a1466587" is_pre_election: true
I20260927 04:58:59.865057 22737 raft_consensus.cc:2454] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 0.
I20260927 04:58:59.865111 22455 raft_consensus.cc:2454] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 0.
I20260927 04:58:59.865578 22522 leader_election.cc:302] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3f19266f53435e8cb2eeda3684c4f5, d9d20f78667b4d32aa477221a5daefd7; no voters:
I20260927 04:58:59.865933 22809 raft_consensus.cc:2781] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:58:59.866083 22809 raft_consensus.cc:482] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:58:59.866341 22809 raft_consensus.cc:3037] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.870882 22809 raft_consensus.cc:504] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.872327 22809 leader_election.cc:288] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 election: Requested vote from peers eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581), 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075)
I20260927 04:58:59.872488 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d11be79a3042aa99ccea0aa80e2a84" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3f19266f53435e8cb2eeda3684c4f5"
I20260927 04:58:59.872881 22737 raft_consensus.cc:3037] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.875036 22737 raft_consensus.cc:2454] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 1.
I20260927 04:58:59.875643 22522 leader_election.cc:302] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3f19266f53435e8cb2eeda3684c4f5, d9d20f78667b4d32aa477221a5daefd7; no voters:
I20260927 04:58:59.875934 22809 raft_consensus.cc:2781] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:58:59.876999 22809 raft_consensus.cc:686] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [term 1 LEADER]: Becoming Leader. State: Replica: d9d20f78667b4d32aa477221a5daefd7, State: Running, Role: LEADER
I20260927 04:58:59.877306 22809 consensus_queue.cc:237] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } }
I20260927 04:58:59.878714 22455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d11be79a3042aa99ccea0aa80e2a84" candidate_uuid: "d9d20f78667b4d32aa477221a5daefd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb02f270efd54335a6a880e9a1466587"
I20260927 04:58:59.879099 22455 raft_consensus.cc:3037] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:58:59.880167 22844 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: TxnStatusTablet state changed. Reason: New leader d9d20f78667b4d32aa477221a5daefd7. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.880611 22844 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:58:59.883668 22455 raft_consensus.cc:2454] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d20f78667b4d32aa477221a5daefd7 in term 1.
I20260927 04:58:59.883396 22293 catalog_manager.cc:5495] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66). New cstate: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } health_report { overall_health: UNKNOWN } } }
I20260927 04:58:59.884730 22847 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 04:58:59.885218 22847 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 04:58:59.941185 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.932914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.20.169.66:45193)
0927 04:58:59.933208 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.933215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.933240 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:58:59.933342 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.933357 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.933446 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.933555 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:58:59.934364 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.935361 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.938178 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.938855 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.938954 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.940289 (+ 1335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:58:59.940408 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:58:59.940483 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.940862 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.940925 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 04:58:59.941537 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.932682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45193 (local address 127.0.0.1:55706)
0927 04:58:59.932980 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 04:58:59.933004 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 04:58:59.933142 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:58:59.933856 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:58:59.933861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:58:59.933876 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:58:59.934191 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.934204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.935543 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.935558 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.937957 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:58:59.937975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.939109 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.939126 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:58:59.939727 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.939931 (+ 204us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:58:59.940596 (+ 665us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:58:59.940672 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 04:58:59.940787 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 04:58:59.941007 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 04:58:59.946534 22455 raft_consensus.cc:1261] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Refusing update from remote peer d9d20f78667b4d32aa477221a5daefd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:58:59.946645 22737 raft_consensus.cc:1261] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Refusing update from remote peer d9d20f78667b4d32aa477221a5daefd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:58:59.947373 22844 consensus_queue.cc:1035] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:58:59.947994 22809 consensus_queue.cc:1035] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:58:59.955199 22808 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: TxnStatusTablet state changed. Reason: New leader d9d20f78667b4d32aa477221a5daefd7. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.955626 22808 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.957054 22810 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: TxnStatusTablet state changed. Reason: New leader d9d20f78667b4d32aa477221a5daefd7. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.957468 22810 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.960919 22809 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.961859 22809 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:58:59.961798 22850 mvcc.cc:205] Tried to move back new op lower bound from 7333827133213798400 to 7333827132941041664. Current Snapshot: MvccSnapshot[applied={T|T < 7333827133213798400}]
I20260927 04:58:59.962114 22809 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.963057 22809 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P d9d20f78667b4d32aa477221a5daefd7: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:58:59.974258 22808 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.974591 22808 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P eb02f270efd54335a6a880e9a1466587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.980249 22810 tablet_replica.cc:439] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9d20f78667b4d32aa477221a5daefd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 } } peers { permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 } } peers { permanent_uuid: "7c3f19266f53435e8cb2eeda3684c4f5" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 39075 } } }
I20260927 04:58:59.980723 22810 tablet_replica.cc:442] T a2d11be79a3042aa99ccea0aa80e2a84 P 7c3f19266f53435e8cb2eeda3684c4f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:58:59.992667 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:58:59.982882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.20.169.67:39075)
0927 04:58:59.984983 (+ 2101us) server_negotiation.cc:206] Beginning negotiation
0927 04:58:59.984988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:58:59.985004 (+ 16us) server_negotiation.cc:407] Connection header received
0927 04:58:59.985084 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:58:59.985087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:58:59.985159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:58:59.985376 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:58:59.986178 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.987327 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.989464 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:58:59.990152 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:58:59.990318 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:58:59.990918 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:58:59.991612 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:58:59.991775 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 04:58:59.992068 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 04:58:59.992147 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1953,"thread_start_us":1748,"threads_started":1}
I20260927 04:59:00.017153 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.007150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:34273 (local address 127.20.169.66:45193)
0927 04:59:00.007424 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.007432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.008541 (+ 1109us) server_negotiation.cc:407] Connection header received
0927 04:59:00.008799 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.008804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.008930 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.009026 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:00.010164 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.011566 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.014244 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.015270 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.015400 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.016079 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:00.016336 (+ 257us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:00.016428 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.016613 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.016736 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 04:59:00.017627 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.007438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45193 (local address 127.20.169.67:34273)
0927 04:59:00.008421 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:00.008443 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:00.008636 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:00.009066 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:00.009072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:00.009088 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:00.009966 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:00.009982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.011715 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.011721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:00.014039 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:00.014067 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.015532 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.015539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:00.015771 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.015854 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:00.016447 (+ 593us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:00.016458 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 04:59:00.016541 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 04:59:00.016690 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":569,"threads_started":1}
I20260927 04:59:00.033200 22455 raft_consensus.cc:1261] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Refusing update from remote peer 7c3f19266f53435e8cb2eeda3684c4f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:00.033432 22591 raft_consensus.cc:1261] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Refusing update from remote peer 7c3f19266f53435e8cb2eeda3684c4f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:00.034204 22832 consensus_queue.cc:1035] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb02f270efd54335a6a880e9a1466587" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 04:59:00.035470 22810 consensus_queue.cc:1035] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d20f78667b4d32aa477221a5daefd7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:00.053506 22851 mvcc.cc:205] Tried to move back new op lower bound from 7333827133567660032 to 7333827131893207040. Current Snapshot: MvccSnapshot[applied={T|T < 7333827133567660032}]
I20260927 04:59:00.119930 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.102988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:37037 (local address 127.20.169.67:39075)
0927 04:59:00.103175 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.103184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.112050 (+ 8866us) server_negotiation.cc:407] Connection header received
0927 04:59:00.112420 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.112428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.112528 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.112622 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:00.114100 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.115080 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.117500 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.118143 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.118305 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.118985 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:00.119094 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:00.119245 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.119397 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.119478 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:59:00.120285 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.100284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:39075 (local address 127.20.169.66:37037)
0927 04:59:00.111945 (+ 11661us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:00.111967 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:00.112191 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:00.112928 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:00.112933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:00.112946 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:00.113846 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:00.113860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.115230 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.115238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:00.117344 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:00.117359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.118464 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.118470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:00.118669 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.118730 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:00.119243 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:00.119258 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 04:59:00.119360 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:59:00.119511 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11349,"thread_start_us":10213,"threads_started":1}
I20260927 04:59:00.164656 22293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:00.165081 22293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:00.170008 22293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1dc70c2e95a34938a092c16e647e922b: Sending DeleteTablet for 3 replicas of tablet 142003ed2ac644929849befb44ca23a8
I20260927 04:59:00.171339 22435 tablet_service.cc:1514] Processing DeleteTablet for tablet 142003ed2ac644929849befb44ca23a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:00 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:33666
I20260927 04:59:00.171679 22570 tablet_service.cc:1514] Processing DeleteTablet for tablet 142003ed2ac644929849befb44ca23a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:00 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:55688
I20260927 04:59:00.175328 22711 tablet_service.cc:1514] Processing DeleteTablet for tablet 142003ed2ac644929849befb44ca23a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:00 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53508
I20260927 04:59:00.177619 22874 tablet_replica.cc:330] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: stopping tablet replica
I20260927 04:59:00.178992 22874 raft_consensus.cc:2229] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:00.180322 22874 raft_consensus.cc:2258] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:00.182680 22874 ts_tablet_manager.cc:1905] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:00.191109 22871 tablet_replica.cc:330] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: stopping tablet replica
I20260927 04:59:00.191449 22871 raft_consensus.cc:2229] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:00.191804 22871 raft_consensus.cc:2258] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:00.193197 22871 ts_tablet_manager.cc:1905] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:00.203452 22871 ts_tablet_manager.cc:1918] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:00.203655 22871 log.cc:1196] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/142003ed2ac644929849befb44ca23a8
I20260927 04:59:00.204281 22871 ts_tablet_manager.cc:1939] T 142003ed2ac644929849befb44ca23a8 P eb02f270efd54335a6a880e9a1466587: Deleting consensus metadata
I20260927 04:59:00.205600 22279 catalog_manager.cc:4841] TS eb02f270efd54335a6a880e9a1466587 (127.20.169.65:39581): tablet 142003ed2ac644929849befb44ca23a8 (table test-table [id=004f6702adb3496dbff24c3267f44772]) successfully deleted
I20260927 04:59:00.208256 22872 tablet_replica.cc:330] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: stopping tablet replica
I20260927 04:59:00.208540 22872 raft_consensus.cc:2229] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:00.208617 22874 ts_tablet_manager.cc:1918] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:00.208858 22874 log.cc:1196] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/142003ed2ac644929849befb44ca23a8
I20260927 04:59:00.209393 22872 raft_consensus.cc:2258] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:00.209658 22874 ts_tablet_manager.cc:1939] T 142003ed2ac644929849befb44ca23a8 P 7c3f19266f53435e8cb2eeda3684c4f5: Deleting consensus metadata
I20260927 04:59:00.211054 22278 catalog_manager.cc:4841] TS 7c3f19266f53435e8cb2eeda3684c4f5 (127.20.169.67:39075): tablet 142003ed2ac644929849befb44ca23a8 (table test-table [id=004f6702adb3496dbff24c3267f44772]) successfully deleted
I20260927 04:59:00.211609 22872 ts_tablet_manager.cc:1905] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:00.221840 22872 ts_tablet_manager.cc:1918] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:00.222044 22872 log.cc:1196] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/142003ed2ac644929849befb44ca23a8
Sep 27 04:59:00 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:00.222955 22872 ts_tablet_manager.cc:1939] T 142003ed2ac644929849befb44ca23a8 P d9d20f78667b4d32aa477221a5daefd7: Deleting consensus metadata
I20260927 04:59:00.224362 22279 catalog_manager.cc:4841] TS d9d20f78667b4d32aa477221a5daefd7 (127.20.169.66:45193): tablet 142003ed2ac644929849befb44ca23a8 (table test-table [id=004f6702adb3496dbff24c3267f44772]) successfully deleted
I20260927 04:59:00.234218 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.186655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.20.169.65:39581)
0927 04:59:00.201767 (+ 15112us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.201774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.201826 (+ 52us) server_negotiation.cc:407] Connection header received
0927 04:59:00.201907 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.201913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.202044 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.202201 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:00.209181 (+ 6980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.211946 (+ 2765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.214675 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.215531 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.215700 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.224288 (+ 8588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:00.224306 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:00.224326 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:00.224419 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:00.229710 (+ 5291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.230885 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.230895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.230899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.231128 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.231659 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.231665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.231669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.232126 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:00.232445 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.233117 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.233672 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14865,"thread_start_us":14618,"threads_started":1}
W20260927 04:59:00.235457 22495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682
I20260927 04:59:00.275691 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.239015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.20.169.126:43165)
0927 04:59:00.253760 (+ 14745us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.253767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.253788 (+ 21us) server_negotiation.cc:407] Connection header received
0927 04:59:00.253852 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.253858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.253959 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.254153 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:00.256186 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.257644 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.259883 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.260988 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.261164 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.265558 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:00.265575 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:00.265580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:00.265625 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:00.271515 (+ 5890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.272563 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.272568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.272572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.272848 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.273342 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.273347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.273351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.273815 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:00.274207 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.274556 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.274859 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14575,"thread_start_us":14049,"threads_started":1}
W20260927 04:59:00.276588 22293 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43814
I20260927 04:59:00.279346 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22364
I20260927 04:59:00.312175 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.281656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.20.169.65:39581)
0927 04:59:00.281837 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.281843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.292429 (+ 10586us) server_negotiation.cc:407] Connection header received
0927 04:59:00.292677 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.292683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.292774 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.292914 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:00.294851 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.296122 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.298269 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.299138 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.299258 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.303383 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:00.303390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:00.303396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:00.303437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:00.308467 (+ 5030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.309430 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.309436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.309439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.309661 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.310195 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.310200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.310203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.310566 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:00.310916 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.311338 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.311624 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 04:59:00.313395 22495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33688
W20260927 04:59:00.314253 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22364: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:00.314513 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 22364
I20260927 04:59:00.339462 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22505
Sep 27 04:59:00 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:00.360868 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.340359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.20.169.66:45193)
0927 04:59:00.340559 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.340566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.340620 (+ 54us) server_negotiation.cc:407] Connection header received
0927 04:59:00.340865 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.340873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.340992 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.341114 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:00.343169 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.344354 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.346843 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.347608 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.347768 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.353362 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:00.353381 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:00.353405 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:00.353457 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:00.357527 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.358382 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.358391 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.358395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.358563 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.359015 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.359020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.359024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.359478 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:00.359780 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.360092 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.360318 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 04:59:00.361677 22637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55716
W20260927 04:59:00.363284 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22505: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:00.363482 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 22505
I20260927 04:59:00.391883 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22646
Sep 27 04:59:00 dist-test-slave-39f2 krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:00.413507 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:00.393015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.20.169.67:39075)
0927 04:59:00.393209 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:00.393215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:00.394144 (+ 929us) server_negotiation.cc:407] Connection header received
0927 04:59:00.394350 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:00.394355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:00.394470 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:00.394588 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:00.396310 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.397508 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.399422 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:00.400096 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:00.400215 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:00.406171 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:00.406192 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:00.406213 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:00.406289 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:00.410439 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.411120 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.411132 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.411136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.411336 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:00.411680 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:00.411686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:00.411690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:00.412174 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:00.412516 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:00.412687 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 04:59:00.412986 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 04:59:00.414318 22777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53526
W20260927 04:59:00.414983 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:00.415156 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 22646
I20260927 04:59:00.435416 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22261
I20260927 04:59:00.436606 22333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:00.549489 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 22261
2026-09-27T04:59:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4989 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:00 dist-test-slave-39f2 krb5kdc[22884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:00 dist-test-slave-39f2 krb5kdc[22884](info): set up 2 sockets
Sep 27 04:59:00 dist-test-slave-39f2 krb5kdc[22884](info): commencing operation
krb5kdc: starting...
W20260927 04:59:01.644699 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:01 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:01.842547 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38515
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:02.163394 22897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:02.163754 22897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:02.163918 22897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:02.171208 22897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:02.171362 22897 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 04:59:02.171463 22897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:02.171579 22897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:02.171674 22897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:02.171761 22897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:02.182204 22897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:38515
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:02.183527 22897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:02.184882 22897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:02.196669 22902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:02.196661 22905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:02.196728 22903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:02.197571 22897 server_base.cc:1034] running on GCE node
I20260927 04:59:02.198367 22897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:02.199656 22897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:02.200932 22897 hybrid_clock.cc:648] HybridClock initialized: now 1790485142200849 us; error 56 us; skew 500 ppm
I20260927 04:59:02.201216 22897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:02 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:02.204392 22897 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:02.206036 22897 webserver.cc:460] Webserver started at http://127.20.169.126:41325/ using document root <none> and password file <none>
I20260927 04:59:02.206727 22897 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:02.206813 22897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:02.207171 22897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:02.210321 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "9bcaf291e1e34ed89410b76ee13fd63e"
format_stamp: "Formatted at 2026-09-27 04:59:02 on dist-test-slave-39f2"
server_key: "35ec18370c8e4d5f78664a6d636a28d7"
server_key_iv: "7533f20f05a296edd52573e022a92eb2"
server_key_version: "encryptionkey@0"
I20260927 04:59:02.211126 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "9bcaf291e1e34ed89410b76ee13fd63e"
format_stamp: "Formatted at 2026-09-27 04:59:02 on dist-test-slave-39f2"
server_key: "35ec18370c8e4d5f78664a6d636a28d7"
server_key_iv: "7533f20f05a296edd52573e022a92eb2"
server_key_version: "encryptionkey@0"
I20260927 04:59:02.216646 22897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 04:59:02.220223 22912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:02.221745 22897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:02.221943 22897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "9bcaf291e1e34ed89410b76ee13fd63e"
format_stamp: "Formatted at 2026-09-27 04:59:02 on dist-test-slave-39f2"
server_key: "35ec18370c8e4d5f78664a6d636a28d7"
server_key_iv: "7533f20f05a296edd52573e022a92eb2"
server_key_version: "encryptionkey@0"
I20260927 04:59:02.222159 22897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:02.234126 22897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:02.238329 22897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:02.238592 22897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:02.260833 22897 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:38515
I20260927 04:59:02.260955 22973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:38515 every 8 connection(s)
I20260927 04:59:02.262218 22897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:02.266145 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:02.271478 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 22897
I20260927 04:59:02.271684 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:02.272456 21157 external_mini_cluster.cc:1428] Setting key 1fc6321d26a46775524c6047494002fd
I20260927 04:59:02.277175 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Bootstrap starting.
I20260927 04:59:02.281028 22974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:02.282259 22974 log.cc:824] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:02.285605 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: No bootstrap required, opened a new log
Sep 27 04:59:02 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:02.294836 22974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38515 } }
I20260927 04:59:02.295254 22974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:02.295379 22974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcaf291e1e34ed89410b76ee13fd63e, State: Initialized, Role: FOLLOWER
I20260927 04:59:02.296067 22974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38515 } }
I20260927 04:59:02.296373 22974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:02.296520 22974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:02.296703 22974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:02.298080 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:02.274616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.20.169.126:38515)
0927 04:59:02.275441 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:02.275455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:02.275502 (+ 47us) server_negotiation.cc:407] Connection header received
0927 04:59:02.277839 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:02.277876 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:02.278337 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:02.278748 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:02.280198 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.282321 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.284163 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.285218 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.285400 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:02.289339 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:02.289350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:02.289368 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:02.289421 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:02.293464 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:02.294086 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:02.294093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:02.294102 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:02.294319 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:02.294653 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:02.294657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:02.294660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:02.295379 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:02.295664 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:02.296168 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 04:59:02.296633 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":333,"threads_started":1}
I20260927 04:59:02.299546 22974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38515 } }
I20260927 04:59:02.300122 22974 leader_election.cc:302] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bcaf291e1e34ed89410b76ee13fd63e; no voters:
I20260927 04:59:02.300772 22974 leader_election.cc:288] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:02.301002 22979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:02.302146 22979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [term 1 LEADER]: Becoming Leader. State: Replica: 9bcaf291e1e34ed89410b76ee13fd63e, State: Running, Role: LEADER
I20260927 04:59:02.302678 22979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38515 } }
I20260927 04:59:02.302845 22974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:02.306352 22981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bcaf291e1e34ed89410b76ee13fd63e. Latest consensus state: current_term: 1 leader_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38515 } } }
I20260927 04:59:02.306704 22981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:02.306211 22980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bcaf291e1e34ed89410b76ee13fd63e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38515 } } }
I20260927 04:59:02.307120 22980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:02.307832 22987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:02.314342 22987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:02.324102 22987 catalog_manager.cc:1347] Generated new cluster ID: 5442c5290af048fb9bf437fd192d74cb
I20260927 04:59:02.324204 22987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:02.339323 22987 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:02.341362 22987 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:02.349745 22987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Generated new TSK 0
I20260927 04:59:02.350809 22987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:02.404333 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38515
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:02.732838 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:02.733168 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:02.733348 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:02.740464 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:02.740613 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:02.740772 23001 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 04:59:02.740881 23001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:59:02.741008 23001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:02.750574 23001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:02.751924 23001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:02.753350 23001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:02.766633 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:02.766721 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:02.766721 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:02.767473 23001 server_base.cc:1034] running on GCE node
I20260927 04:59:02.768139 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:02.769177 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:02.770414 23001 hybrid_clock.cc:648] HybridClock initialized: now 1790485142770371 us; error 52 us; skew 500 ppm
I20260927 04:59:02.770743 23001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:02 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:02.774039 23001 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:02.775491 23001 webserver.cc:460] Webserver started at http://127.20.169.65:45957/ using document root <none> and password file <none>
I20260927 04:59:02.776124 23001 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:02.776221 23001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:02.776484 23001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:02.779415 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "59d88268997847f8a5cc96f0421e77b3"
format_stamp: "Formatted at 2026-09-27 04:59:02 on dist-test-slave-39f2"
server_key: "51b1c8942d734da9d16cfc64692271d2"
server_key_iv: "d0de31d1b32074fd27572bce830f407c"
server_key_version: "encryptionkey@0"
I20260927 04:59:02.780177 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "59d88268997847f8a5cc96f0421e77b3"
format_stamp: "Formatted at 2026-09-27 04:59:02 on dist-test-slave-39f2"
server_key: "51b1c8942d734da9d16cfc64692271d2"
server_key_iv: "d0de31d1b32074fd27572bce830f407c"
server_key_version: "encryptionkey@0"
I20260927 04:59:02.785319 23001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 04:59:02.788659 23016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:02.789970 23001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:02.790107 23001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "59d88268997847f8a5cc96f0421e77b3"
format_stamp: "Formatted at 2026-09-27 04:59:02 on dist-test-slave-39f2"
server_key: "51b1c8942d734da9d16cfc64692271d2"
server_key_iv: "d0de31d1b32074fd27572bce830f407c"
server_key_version: "encryptionkey@0"
I20260927 04:59:02.790262 23001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:02.813920 23001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:02.817978 23001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:02.818234 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:02.820847 23001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:02.820976 23001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:02.821086 23001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:02.821170 23001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:59:02 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:02.846983 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:02.821527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:51241 (local address 127.20.169.126:38515)
0927 04:59:02.822022 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:02.822031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:02.823449 (+ 1418us) server_negotiation.cc:407] Connection header received
0927 04:59:02.825751 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:02.825757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:02.825866 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:02.826074 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:02.829026 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.830407 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.833034 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.833921 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.834091 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:02.839543 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:02.839552 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:02.839560 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:02.839610 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:02.842916 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:02.844027 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:02.844034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:02.844039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:02.844190 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:02.844877 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:02.844882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:02.844885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:02.845304 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:02.845449 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:02.846391 (+ 942us) server_negotiation.cc:299] Negotiation successful
0927 04:59:02.846604 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":255,"threads_started":1}
I20260927 04:59:02.848519 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:02.822333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.20.169.65:51241)
0927 04:59:02.823220 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:02.823259 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:02.825410 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:02.826400 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:02.826417 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:02.827428 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:02.828683 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:02.828736 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.830617 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.830626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:02.832858 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:02.832879 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.834172 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.834177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:02.834370 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:02.835200 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:02.835240 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:02.839239 (+ 3999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:02.843123 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:02.843132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:02.843154 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:02.843807 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:02.844318 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:02.844327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:02.844331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:02.844673 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:02.845435 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:02.845442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:02.846016 (+ 574us) client_negotiation.cc:769] Sending connection context
0927 04:59:02.846308 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 04:59:02.846851 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":414,"threads_started":1}
I20260927 04:59:02.866531 23001 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:41679
I20260927 04:59:02.866856 23136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:41679 every 8 connection(s)
I20260927 04:59:02.868093 23001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:02.870534 23137 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38515
I20260927 04:59:02.870774 23137 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:02.871182 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:02.855336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:50027 (local address 127.20.169.126:38515)
0927 04:59:02.855518 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:02.855524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:02.856393 (+ 869us) server_negotiation.cc:407] Connection header received
0927 04:59:02.856700 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:02.856711 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:02.856770 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:02.856920 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:02.858087 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.859277 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.861351 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.862158 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.862312 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:02.864778 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:02.864784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:02.864792 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:02.864873 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:02.868268 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:02.869161 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:02.869166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:02.869169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:02.869326 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:02.869763 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:02.869769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:02.869772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:02.870106 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:02.870306 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:02.870750 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 04:59:02.870964 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:59:02.871404 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:02.855587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.20.169.65:50027)
0927 04:59:02.856268 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:02.856291 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:02.856514 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:02.857135 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:02.857143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:02.857600 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:02.857939 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:02.857949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.859463 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.859473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:02.861186 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:02.861206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:02.862391 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:02.862405 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:02.862628 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:02.863364 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:02.863397 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:02.864610 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:02.868420 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:02.868423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:02.868426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:02.869004 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:02.869473 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:02.869479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:02.869485 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:02.869647 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:02.870243 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:02.870247 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:02.870432 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 04:59:02.870619 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 04:59:02.870950 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":378,"threads_started":1}
I20260927 04:59:02.871784 23137 heartbeater.cc:511] Master 127.20.169.126:38515 requested a full tablet report, sending...
I20260927 04:59:02.873682 22929 ts_manager.cc:194] Registered new tserver with Master: 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679)
I20260927 04:59:02.875061 22929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:51241
I20260927 04:59:02.875686 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 23001
I20260927 04:59:02.875916 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:02.876729 21157 external_mini_cluster.cc:1428] Setting key 7b9be2be07596783fb46d64e43085bf8
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:02.919988 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38515
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:03.247006 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:03.247334 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:03.247519 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:03.254711 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:03.254853 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:03.255007 23142 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 04:59:03.255092 23142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:59:03.255234 23142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:03.265044 23142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:03.266286 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:03.267562 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:03.280586 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:03.280618 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:03.281075 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:03.281484 23142 server_base.cc:1034] running on GCE node
I20260927 04:59:03.282114 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:03.283027 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:03.284205 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790485143284142 us; error 55 us; skew 500 ppm
I20260927 04:59:03.284533 23142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:03 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:03.292606 23142 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:03.294545 23142 webserver.cc:460] Webserver started at http://127.20.169.66:40159/ using document root <none> and password file <none>
I20260927 04:59:03.295239 23142 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:03.295388 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:03.295718 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:03.298732 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "f0012c42fac845918cc8629b01cac45f"
format_stamp: "Formatted at 2026-09-27 04:59:03 on dist-test-slave-39f2"
server_key: "b94d8bfc3db76135280fd9c651d4672c"
server_key_iv: "379d99fc1d5e6cb5538d850b77844a80"
server_key_version: "encryptionkey@0"
I20260927 04:59:03.299521 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "f0012c42fac845918cc8629b01cac45f"
format_stamp: "Formatted at 2026-09-27 04:59:03 on dist-test-slave-39f2"
server_key: "b94d8bfc3db76135280fd9c651d4672c"
server_key_iv: "379d99fc1d5e6cb5538d850b77844a80"
server_key_version: "encryptionkey@0"
I20260927 04:59:03.305035 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 04:59:03.308619 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:03.310137 23142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:03.310324 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "f0012c42fac845918cc8629b01cac45f"
format_stamp: "Formatted at 2026-09-27 04:59:03 on dist-test-slave-39f2"
server_key: "b94d8bfc3db76135280fd9c651d4672c"
server_key_iv: "379d99fc1d5e6cb5538d850b77844a80"
server_key_version: "encryptionkey@0"
I20260927 04:59:03.310535 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:03.323473 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:03.327549 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:03.327841 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:03.330669 23142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:03.330811 23142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:03.330965 23142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:03.331061 23142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:59:03 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:03.354713 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.331192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:56815 (local address 127.20.169.126:38515)
0927 04:59:03.331388 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:03.331396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:03.332776 (+ 1380us) server_negotiation.cc:407] Connection header received
0927 04:59:03.334985 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:03.334992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:03.335085 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:03.335198 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:03.337667 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.339075 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.341427 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.342236 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.342414 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.346367 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:03.346372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:03.346375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:03.346434 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:03.350277 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.351293 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.351303 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.351309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.351533 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.352175 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.352180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.352184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.352631 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:03.352792 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:03.353956 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0927 04:59:03.354233 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:59:03.356319 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.331859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.20.169.66:56815)
0927 04:59:03.332622 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:03.332657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:03.334616 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:03.335514 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:03.335530 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:03.336422 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:03.337336 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.337359 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.339286 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.339293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.341267 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.341279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.342461 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.342466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.342648 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.343344 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:03.343366 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:03.346119 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:03.350428 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.350440 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.350454 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.351115 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.351680 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.351687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.351693 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.352009 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.352796 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:03.352833 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:03.353411 (+ 578us) client_negotiation.cc:769] Sending connection context
0927 04:59:03.353796 (+ 385us) client_negotiation.cc:240] Negotiation successful
0927 04:59:03.354505 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":332,"threads_started":1}
I20260927 04:59:03.377751 23142 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:35957
I20260927 04:59:03.377775 23276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:35957 every 8 connection(s)
I20260927 04:59:03.379284 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:03.381991 23277 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38515
I20260927 04:59:03.382234 23277 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:03.383044 23277 heartbeater.cc:511] Master 127.20.169.126:38515 requested a full tablet report, sending...
I20260927 04:59:03.384698 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.364709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:53307 (local address 127.20.169.126:38515)
0927 04:59:03.364962 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:03.364971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:03.369716 (+ 4745us) server_negotiation.cc:407] Connection header received
0927 04:59:03.370050 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:03.370057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:03.370148 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:03.370267 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:03.371193 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.372442 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.375161 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.375725 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.375822 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.378525 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:03.378529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:03.378532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:03.378563 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:03.381575 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.382549 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.382554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.382557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.382722 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.383143 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.383148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.383150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.383588 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:03.383739 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:03.384282 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 04:59:03.384477 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:59:03.384841 22929 ts_manager.cc:194] Registered new tserver with Master: f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957)
I20260927 04:59:03.385035 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.364732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.20.169.66:53307)
0927 04:59:03.369566 (+ 4834us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:03.369594 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:03.369802 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:03.370309 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:03.370314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:03.370757 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:03.371043 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.371052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.372627 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.372636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.375014 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.375029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.375925 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.375935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.376112 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.376990 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:03.377044 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:03.378335 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:03.381717 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.381725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.381732 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.382412 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.382853 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.382857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.382860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.383006 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.383716 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:03.383719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:03.383945 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 04:59:03.384128 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 04:59:03.384470 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4618,"thread_start_us":4417,"threads_started":1}
I20260927 04:59:03.386062 22929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:56815
I20260927 04:59:03.389537 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 23142
I20260927 04:59:03.389719 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:03.390194 21157 external_mini_cluster.cc:1428] Setting key 9367a1d6179d4b1f0225f3ec7bfe4d06
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:03.432475 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38515
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:03.764415 23281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:03.764672 23281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:03.764771 23281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:03.771970 23281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:03.772078 23281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:03.772161 23281 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 04:59:03.772212 23281 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:59:03.772274 23281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:03.782114 23281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:03.783303 23281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:03.784569 23281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:03.797595 23286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:03.797600 23289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:03.797835 23287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:03.799271 23281 server_base.cc:1034] running on GCE node
I20260927 04:59:03.799980 23281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:03.801038 23281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:03.802320 23281 hybrid_clock.cc:648] HybridClock initialized: now 1790485143802206 us; error 55 us; skew 500 ppm
I20260927 04:59:03.802704 23281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:03 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:03.806461 23281 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:03.807930 23281 webserver.cc:460] Webserver started at http://127.20.169.67:41753/ using document root <none> and password file <none>
I20260927 04:59:03.808647 23281 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:03.808831 23281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:03.809141 23281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:03.812053 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "3934a2395abf4a5ab41b6c9bb53bcab7"
format_stamp: "Formatted at 2026-09-27 04:59:03 on dist-test-slave-39f2"
server_key: "31813cf439f7fa41cbe7791275f76f42"
server_key_iv: "f9f6d4de4bab3346efa714fee653f48f"
server_key_version: "encryptionkey@0"
I20260927 04:59:03.812932 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "3934a2395abf4a5ab41b6c9bb53bcab7"
format_stamp: "Formatted at 2026-09-27 04:59:03 on dist-test-slave-39f2"
server_key: "31813cf439f7fa41cbe7791275f76f42"
server_key_iv: "f9f6d4de4bab3346efa714fee653f48f"
server_key_version: "encryptionkey@0"
I20260927 04:59:03.818150 23281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 04:59:03.821492 23296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:03.822731 23281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:03.822901 23281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "3934a2395abf4a5ab41b6c9bb53bcab7"
format_stamp: "Formatted at 2026-09-27 04:59:03 on dist-test-slave-39f2"
server_key: "31813cf439f7fa41cbe7791275f76f42"
server_key_iv: "f9f6d4de4bab3346efa714fee653f48f"
server_key_version: "encryptionkey@0"
I20260927 04:59:03.823092 23281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:03.839594 23281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:03.843955 23281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:03.844295 23281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:03.847184 23281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:03.847322 23281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:03.847461 23281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:03.847570 23281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:59:03 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485143, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:03.874449 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.847700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:58535 (local address 127.20.169.126:38515)
0927 04:59:03.847886 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:03.847895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:03.849401 (+ 1506us) server_negotiation.cc:407] Connection header received
0927 04:59:03.851445 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:03.851451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:03.851540 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:03.851652 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:03.853496 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.854975 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.858571 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.859180 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.859319 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.864797 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:03.864802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:03.864805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:03.864909 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:03.869140 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.870535 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.870542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.870547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.870739 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.871347 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.871351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.871355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.871757 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:03.871891 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:03.873854 (+ 1963us) server_negotiation.cc:299] Negotiation successful
0927 04:59:03.874099 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:59:03.875994 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.848416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.20.169.67:58535)
0927 04:59:03.849238 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:03.849275 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:03.851116 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:03.851898 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:03.851905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:03.852417 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:03.853266 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.853297 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.855248 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.855259 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.858381 (+ 3122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.858405 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.859673 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.859683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.860034 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.860821 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:03.860864 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:03.864513 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:03.869383 (+ 4870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.869392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.869435 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.870321 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.870905 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.870910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.870915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.871175 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.871970 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:03.871978 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:03.872435 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 04:59:03.872720 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 04:59:03.873293 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":386,"threads_started":1}
I20260927 04:59:03.878123 23137 heartbeater.cc:503] Master 127.20.169.126:38515 was elected leader, sending a full tablet report...
I20260927 04:59:03.900774 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.883929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:51225 (local address 127.20.169.126:38515)
0927 04:59:03.884135 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:03.884143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:03.885132 (+ 989us) server_negotiation.cc:407] Connection header received
0927 04:59:03.885412 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:03.885420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:03.885545 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:03.885632 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:03.887203 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.888250 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.890079 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.891081 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.891265 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.893181 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:03.893188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:03.893194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:03.893229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:03.897587 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.898299 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.898308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.898314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.898494 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.899104 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.899113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.899116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.899548 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:03.899727 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:03.900213 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 04:59:03.900426 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 04:59:03.900890 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.884111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.20.169.67:51225)
0927 04:59:03.884992 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:03.885022 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:03.885185 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:03.885863 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:03.885867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:03.886527 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:03.887048 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.887068 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.888366 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.888376 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.889862 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:03.889881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.891271 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.891275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:03.891413 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.892161 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:03.892177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:03.892978 (+ 801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:03.897741 (+ 4763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.897745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.897747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.898174 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.898668 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:03.898682 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:03.898689 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:03.898917 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:03.899702 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:03.899708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:03.899945 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 04:59:03.900106 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 04:59:03.900425 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":476,"threads_started":1}
I20260927 04:59:03.917986 23281 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:34649
I20260927 04:59:03.918032 23417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:34649 every 8 connection(s)
I20260927 04:59:03.919350 23281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:03.921573 23418 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38515
I20260927 04:59:03.921766 23418 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:03.922435 23418 heartbeater.cc:511] Master 127.20.169.126:38515 requested a full tablet report, sending...
I20260927 04:59:03.923774 22928 ts_manager.cc:194] Registered new tserver with Master: 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
I20260927 04:59:03.924109 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 23281
I20260927 04:59:03.924301 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:03.924870 21157 external_mini_cluster.cc:1428] Setting key 1bab16de13ddd06be1cd53385fdd4568
I20260927 04:59:03.924891 22928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:58535
I20260927 04:59:03.928396 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:59:03 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:59:03 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:03.988724 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:03.973726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.20.169.126:38515)
0927 04:59:03.973931 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:03.973935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:03.974275 (+ 340us) server_negotiation.cc:407] Connection header received
0927 04:59:03.974482 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:03.974487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:03.974558 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:03.974655 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:03.976198 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.977170 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.978551 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:03.978988 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:03.979089 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:03.983423 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:03.983428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:03.983431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:03.983466 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:03.986111 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.986758 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.986765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.986768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.986905 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:03.987284 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:03.987289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:03.987292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:03.987570 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:03.987823 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:03.988131 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 04:59:03.988356 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 04:59:03.994019 22928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:04.004613 22928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:04.034931 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.018524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:35957 (local address 127.0.0.1:39494)
0927 04:59:04.022204 (+ 3680us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.022224 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.022351 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.023420 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.023428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.023464 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.024342 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.024370 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.026936 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.026945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.030379 (+ 3434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.030404 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.033670 (+ 3266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.033679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.033981 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.034055 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.034213 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.034484 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3545,"thread_start_us":3394,"threads_started":1}
I20260927 04:59:04.035676 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.019065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.20.169.66:35957)
0927 04:59:04.021842 (+ 2777us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.021856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.022331 (+ 475us) server_negotiation.cc:407] Connection header received
0927 04:59:04.022614 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.022626 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.022911 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.023150 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.024640 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.026685 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.030662 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.032996 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.033140 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.033423 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.035205 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.035392 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2548,"thread_start_us":2381,"threads_started":1}
I20260927 04:59:04.037860 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.018360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41679 (local address 127.0.0.1:53408)
0927 04:59:04.019563 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.019621 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.019804 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.023104 (+ 3300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.023117 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.023166 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.025447 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.025467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.030213 (+ 4746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.030222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.035001 (+ 4779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.035018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.036950 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.036956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.037174 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.037219 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.037326 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.037484 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":897,"threads_started":1}
I20260927 04:59:04.038678 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.018631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.20.169.65:41679)
0927 04:59:04.022341 (+ 3710us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.022350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.022376 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:59:04.022510 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.022520 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.022783 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.025774 (+ 2991us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.026068 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.029941 (+ 3873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.035247 (+ 5306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.036508 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.036687 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.036929 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.038233 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.038408 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3522,"thread_start_us":3386,"threads_started":1}
I20260927 04:59:04.039947 23207 tablet_service.cc:1467] Processing CreateTablet for tablet 5650992cfc9b45e1a2d8c67f92cbe204 (DEFAULT_TABLE table=test-table [id=76e1242d6c92436db8b7b754cff40cfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:04.041540 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5650992cfc9b45e1a2d8c67f92cbe204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.041846 23067 tablet_service.cc:1467] Processing CreateTablet for tablet 5650992cfc9b45e1a2d8c67f92cbe204 (DEFAULT_TABLE table=test-table [id=76e1242d6c92436db8b7b754cff40cfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:04.043114 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5650992cfc9b45e1a2d8c67f92cbe204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.044589 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.022514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34649 (local address 127.0.0.1:36752)
0927 04:59:04.034573 (+ 12059us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.034593 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.034710 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.036837 (+ 2127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.036843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.036884 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.037244 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.037256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.039252 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.039258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.041329 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.041348 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.043741 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.043748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.043928 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.043960 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.044041 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.044186 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11934,"thread_start_us":12773,"threads_started":1}
I20260927 04:59:04.045389 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.022661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.20.169.67:34649)
0927 04:59:04.035893 (+ 13232us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.035902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.035940 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:59:04.036047 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.036056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.036312 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.036516 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.037467 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.039095 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.041521 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.043240 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.043377 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.043593 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.044865 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.045018 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12461,"spinlock_wait_cycles":1214848,"thread_start_us":12295,"threads_started":1}
I20260927 04:59:04.049418 23351 tablet_service.cc:1467] Processing CreateTablet for tablet 5650992cfc9b45e1a2d8c67f92cbe204 (DEFAULT_TABLE table=test-table [id=76e1242d6c92436db8b7b754cff40cfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:04.051841 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5650992cfc9b45e1a2d8c67f92cbe204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.055034 23437 tablet_bootstrap.cc:492] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Bootstrap starting.
I20260927 04:59:04.057471 23438 tablet_bootstrap.cc:492] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Bootstrap starting.
I20260927 04:59:04.058717 23437 tablet_bootstrap.cc:654] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.060165 23437 log.cc:824] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:04.061036 23438 tablet_bootstrap.cc:654] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.062484 23438 log.cc:824] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:04.065028 23437 tablet_bootstrap.cc:492] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: No bootstrap required, opened a new log
I20260927 04:59:04.065287 23437 ts_tablet_manager.cc:1397] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 04:59:04.065541 23438 tablet_bootstrap.cc:492] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: No bootstrap required, opened a new log
I20260927 04:59:04.065830 23438 ts_tablet_manager.cc:1397] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 04:59:04.067497 23441 tablet_bootstrap.cc:492] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Bootstrap starting.
I20260927 04:59:04.070953 23441 tablet_bootstrap.cc:654] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.072207 23441 log.cc:824] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:04.075057 23438 raft_consensus.cc:348] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.075464 23438 raft_consensus.cc:374] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.075626 23441 tablet_bootstrap.cc:492] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: No bootstrap required, opened a new log
I20260927 04:59:04.075608 23438 raft_consensus.cc:729] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.075902 23441 ts_tablet_manager.cc:1397] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 04:59:04.075541 23437 raft_consensus.cc:348] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.076376 23437 raft_consensus.cc:374] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.076489 23437 raft_consensus.cc:729] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d88268997847f8a5cc96f0421e77b3, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.076268 23438 consensus_queue.cc:260] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.077200 23437 consensus_queue.cc:260] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.077980 23277 heartbeater.cc:503] Master 127.20.169.126:38515 was elected leader, sending a full tablet report...
I20260927 04:59:04.078316 23438 ts_tablet_manager.cc:1428] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 04:59:04.082121 23437 ts_tablet_manager.cc:1428] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
I20260927 04:59:04.086350 23441 raft_consensus.cc:348] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.086694 23441 raft_consensus.cc:374] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.086786 23441 raft_consensus.cc:729] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.087287 23441 consensus_queue.cc:260] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.088271 23418 heartbeater.cc:503] Master 127.20.169.126:38515 was elected leader, sending a full tablet report...
I20260927 04:59:04.088639 23441 ts_tablet_manager.cc:1428] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260927 04:59:04.123229 23138 tablet.cc:2369] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:04.124752 23443 raft_consensus.cc:482] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:04.125083 23443 raft_consensus.cc:504] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.126919 23443 leader_election.cc:288] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
W20260927 04:59:04.131309 23278 tablet.cc:2369] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:04.134473 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.127371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40495 (local address 127.20.169.65:41679)
0927 04:59:04.127607 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.127613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.128553 (+ 940us) server_negotiation.cc:407] Connection header received
0927 04:59:04.128741 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.128750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.128872 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.128956 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.129610 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.130423 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.132623 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.133538 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.133666 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.133715 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.134096 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.134225 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 04:59:04.134629 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.127397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41679 (local address 127.20.169.66:40495)
0927 04:59:04.128435 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.128457 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.128580 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.129003 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.129008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.129031 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.129469 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.129479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.130564 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.130571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.132474 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.132487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.133751 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.133757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.133930 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.133964 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.134055 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.134192 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":3,"thread_start_us":484,"threads_started":1}
I20260927 04:59:04.135125 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.127370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:35491 (local address 127.20.169.67:34649)
0927 04:59:04.127609 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.127617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.128645 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 04:59:04.128920 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.128924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.129020 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.129120 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.129731 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.130512 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.133008 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.133952 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.134122 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.134187 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.134765 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.134941 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 04:59:04.135505 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.127413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34649 (local address 127.20.169.66:35491)
0927 04:59:04.128550 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.128569 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.128703 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.129153 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.129156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.129188 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.129572 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.129585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.130656 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.130659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.132858 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.132872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.134178 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.134184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.134395 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.134435 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.134580 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.135123 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":62,"thread_start_us":552,"threads_started":1}
I20260927 04:59:04.135600 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5650992cfc9b45e1a2d8c67f92cbe204" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3" is_pre_election: true
I20260927 04:59:04.136255 23087 raft_consensus.cc:2454] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 0.
I20260927 04:59:04.136530 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5650992cfc9b45e1a2d8c67f92cbe204" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" is_pre_election: true
I20260927 04:59:04.136981 23158 leader_election.cc:302] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d88268997847f8a5cc96f0421e77b3, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:04.137022 23371 raft_consensus.cc:2454] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 0.
I20260927 04:59:04.137382 23443 raft_consensus.cc:2781] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:04.137569 23443 raft_consensus.cc:482] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:04.137686 23443 raft_consensus.cc:3037] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Advancing to term 1
W20260927 04:59:04.171708 23419 tablet.cc:2369] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:04.190821 23443 raft_consensus.cc:504] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.191666 23443 leader_election.cc:288] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 election: Requested vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
I20260927 04:59:04.192338 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5650992cfc9b45e1a2d8c67f92cbe204" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7"
I20260927 04:59:04.192345 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5650992cfc9b45e1a2d8c67f92cbe204" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3"
I20260927 04:59:04.192612 23371 raft_consensus.cc:3037] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:04.192606 23087 raft_consensus.cc:3037] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:04.196259 23371 raft_consensus.cc:2454] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 1.
I20260927 04:59:04.196259 23087 raft_consensus.cc:2454] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 1.
I20260927 04:59:04.196851 23160 leader_election.cc:302] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:04.197230 23443 raft_consensus.cc:2781] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:04.197839 23443 raft_consensus.cc:686] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 1 LEADER]: Becoming Leader. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Running, Role: LEADER
I20260927 04:59:04.198361 23443 consensus_queue.cc:237] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.203474 22928 catalog_manager.cc:5495] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f reported cstate change: term changed from 0 to 1, leader changed from <none> to f0012c42fac845918cc8629b01cac45f (127.20.169.66). New cstate: current_term: 1 leader_uuid: "f0012c42fac845918cc8629b01cac45f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } health_report { overall_health: UNKNOWN } } }
Sep 27 04:59:04 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485142, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:04.261471 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.247307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.20.169.126:38515)
0927 04:59:04.247551 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.247562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.248148 (+ 586us) server_negotiation.cc:407] Connection header received
0927 04:59:04.248295 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.248298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.248365 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.248465 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:04.249726 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.250587 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.251967 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.252433 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.252532 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.255981 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:04.255985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:04.255988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:04.256015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:04.258632 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:04.259302 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:04.259309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:04.259315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:04.259451 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:04.259838 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:04.259877 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:04.259881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:04.260150 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:04.260298 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.260791 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.261134 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":109}
I20260927 04:59:04.261492 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.247408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38515 (local address 127.0.0.1:35010)
0927 04:59:04.248041 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.248058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.248185 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.248509 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.248512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.249308 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:04.249570 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.249579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.250716 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.250720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.251779 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.251791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.252609 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.252615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.252736 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.253589 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:04.253606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:04.255802 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:04.258779 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:04.258796 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:04.258807 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:04.259182 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:04.259557 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:04.259563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:04.259566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:04.259733 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:04.260275 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:04.260283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:04.260502 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.260697 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.261003 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":327,"threads_started":1}
I20260927 04:59:04.266595 22928 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:35010:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 04:59:04.267729 22928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:04.274662 23067 tablet_service.cc:1467] Processing CreateTablet for tablet 09bb22255138427fb325999795d37e06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=771ad510068d49318dc6366f2873066f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:59:04.275650 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bb22255138427fb325999795d37e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.275317 23207 tablet_service.cc:1467] Processing CreateTablet for tablet 09bb22255138427fb325999795d37e06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=771ad510068d49318dc6366f2873066f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:59:04.276141 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bb22255138427fb325999795d37e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.278445 23351 tablet_service.cc:1467] Processing CreateTablet for tablet 09bb22255138427fb325999795d37e06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=771ad510068d49318dc6366f2873066f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:59:04.279284 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bb22255138427fb325999795d37e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.282137 23437 tablet_bootstrap.cc:492] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: Bootstrap starting.
I20260927 04:59:04.283514 23437 tablet_bootstrap.cc:654] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.283852 23438 tablet_bootstrap.cc:492] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: Bootstrap starting.
I20260927 04:59:04.285513 23438 tablet_bootstrap.cc:654] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.285904 23437 tablet_bootstrap.cc:492] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: No bootstrap required, opened a new log
I20260927 04:59:04.286046 23437 ts_tablet_manager.cc:1397] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260927 04:59:04.286258 23441 tablet_bootstrap.cc:492] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: Bootstrap starting.
I20260927 04:59:04.286972 23437 raft_consensus.cc:348] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.287216 23437 raft_consensus.cc:374] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.287333 23437 raft_consensus.cc:729] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d88268997847f8a5cc96f0421e77b3, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.287563 23437 consensus_queue.cc:260] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.288035 23441 tablet_bootstrap.cc:654] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.288219 23437 ts_tablet_manager.cc:1428] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:59:04.288055 23444 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.288463 23444 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.289062 23438 tablet_bootstrap.cc:492] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: No bootstrap required, opened a new log
I20260927 04:59:04.289202 23438 ts_tablet_manager.cc:1397] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 04:59:04.290071 23438 raft_consensus.cc:348] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.290349 23438 raft_consensus.cc:374] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.290436 23438 raft_consensus.cc:729] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.290694 23438 consensus_queue.cc:260] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.291138 23443 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.291433 23443 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.292883 23438 ts_tablet_manager.cc:1428] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 04:59:04.294310 23441 tablet_bootstrap.cc:492] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: No bootstrap required, opened a new log
I20260927 04:59:04.294495 23441 ts_tablet_manager.cc:1397] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 04:59:04.295181 23441 raft_consensus.cc:348] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.295401 23441 raft_consensus.cc:374] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.295518 23441 raft_consensus.cc:729] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.295768 23441 consensus_queue.cc:260] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.296367 23441 ts_tablet_manager.cc:1428] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 04:59:04.296221 23445 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.296612 23445 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.470610 23445 raft_consensus.cc:482] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:04.470809 23445 raft_consensus.cc:504] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.472329 23445 leader_election.cc:288] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957)
I20260927 04:59:04.478551 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.472868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:59003 (local address 127.20.169.65:41679)
0927 04:59:04.473125 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.473134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.473599 (+ 465us) server_negotiation.cc:407] Connection header received
0927 04:59:04.473808 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.473812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.473872 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.473966 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.474437 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.475194 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.476781 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.477606 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.477780 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.477869 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.478128 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.478256 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 04:59:04.478636 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.472509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41679 (local address 127.20.169.67:59003)
0927 04:59:04.473485 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.473500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.473655 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.473993 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.473997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.474024 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.474263 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.474276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.475335 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.475343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.476625 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.476637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.477780 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.477787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.477967 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.477992 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.478101 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.478235 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":658,"threads_started":1}
I20260927 04:59:04.479422 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bb22255138427fb325999795d37e06" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3" is_pre_election: true
I20260927 04:59:04.479539 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.472741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:53475 (local address 127.20.169.66:35957)
0927 04:59:04.473022 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.473031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.473737 (+ 706us) server_negotiation.cc:407] Connection header received
0927 04:59:04.473897 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.473903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.473972 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.474076 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:04.474658 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.475646 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.477512 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.478316 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.478493 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.478565 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.479135 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.479268 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 04:59:04.479606 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.472658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:35957 (local address 127.20.169.67:53475)
0927 04:59:04.473629 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.473647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.473781 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.474106 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.474110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.474153 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:04.474501 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.474510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.475800 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.475805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.477356 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.477367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.478516 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.478523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.478917 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.478988 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.479096 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.479224 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":3,"thread_start_us":697,"threads_started":1}
I20260927 04:59:04.479729 23087 raft_consensus.cc:2454] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 0.
I20260927 04:59:04.480264 23297 leader_election.cc:302] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, 59d88268997847f8a5cc96f0421e77b3; no voters:
I20260927 04:59:04.480674 23445 raft_consensus.cc:2781] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:04.480582 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bb22255138427fb325999795d37e06" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0012c42fac845918cc8629b01cac45f" is_pre_election: true
I20260927 04:59:04.480791 23445 raft_consensus.cc:482] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:04.480937 23445 raft_consensus.cc:3037] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:04.481037 23227 raft_consensus.cc:2454] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 0.
I20260927 04:59:04.495682 23445 raft_consensus.cc:504] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.496361 23445 leader_election.cc:288] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 election: Requested vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957)
I20260927 04:59:04.496953 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bb22255138427fb325999795d37e06" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0012c42fac845918cc8629b01cac45f"
I20260927 04:59:04.496937 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bb22255138427fb325999795d37e06" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3"
I20260927 04:59:04.497161 23227 raft_consensus.cc:3037] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:04.497189 23087 raft_consensus.cc:3037] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:04.499311 23227 raft_consensus.cc:2454] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 1.
I20260927 04:59:04.499797 23299 leader_election.cc:302] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:04.500089 23445 raft_consensus.cc:2781] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:04.500638 23087 raft_consensus.cc:2454] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 1.
I20260927 04:59:04.501077 23445 raft_consensus.cc:686] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 LEADER]: Becoming Leader. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Running, Role: LEADER
I20260927 04:59:04.501662 23445 consensus_queue.cc:237] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.597666 23461 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: TxnStatusTablet state changed. Reason: New leader 3934a2395abf4a5ab41b6c9bb53bcab7. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.598090 23461 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:59:04.598790 23463 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 04:59:04.599088 23463 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 04:59:04.599639 22928 catalog_manager.cc:5495] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } health_report { overall_health: HEALTHY } } }
I20260927 04:59:04.660653 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.652937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34649 (local address 127.0.0.1:36768)
0927 04:59:04.653189 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.653211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.653440 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.654470 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.654475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.654490 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:04.654680 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.654691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.655864 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.655869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.657651 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.657667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.658495 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.658502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.658771 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.658833 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.660036 (+ 1203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.660068 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.660165 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.660340 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 04:59:04.660663 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.653194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.20.169.67:34649)
0927 04:59:04.654084 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.654089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.654104 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:59:04.654180 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.654183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.654257 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.654438 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:04.654855 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.655728 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.657816 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.658309 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.658463 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.659065 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.659874 (+ 809us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.660091 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.660209 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.660307 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":622,"threads_started":1}
I20260927 04:59:04.660617 23443 consensus_queue.cc:1035] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:04.671558 23227 raft_consensus.cc:1261] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Refusing update from remote peer 3934a2395abf4a5ab41b6c9bb53bcab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:04.672494 23461 consensus_queue.cc:1035] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:04.679922 23086 raft_consensus.cc:1261] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Refusing update from remote peer 3934a2395abf4a5ab41b6c9bb53bcab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:04.680843 23461 consensus_queue.cc:1035] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:04.694418 23444 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: TxnStatusTablet state changed. Reason: New leader 3934a2395abf4a5ab41b6c9bb53bcab7. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.694759 23444 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.700968 23470 mvcc.cc:205] Tried to move back new op lower bound from 7333827152563896320 to 7333827152258207744. Current Snapshot: MvccSnapshot[applied={T|T < 7333827152563896320}]
I20260927 04:59:04.701764 23448 consensus_queue.cc:1035] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:04.702488 23445 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.702816 23445 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:59:04.703678 23443 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: TxnStatusTablet state changed. Reason: New leader 3934a2395abf4a5ab41b6c9bb53bcab7. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.703915 23443 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.705690 23445 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.706034 23445 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 3934a2395abf4a5ab41b6c9bb53bcab7: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:59:04.708945 23444 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.709182 23444 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P 59d88268997847f8a5cc96f0421e77b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.721948 23443 tablet_replica.cc:439] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } } }
I20260927 04:59:04.722247 23443 tablet_replica.cc:442] T 09bb22255138427fb325999795d37e06 P f0012c42fac845918cc8629b01cac45f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:04.734059 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.726309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.20.169.66:35957)
0927 04:59:04.726598 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.726606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.726719 (+ 113us) server_negotiation.cc:407] Connection header received
0927 04:59:04.727046 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.727052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.727191 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.727314 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:04.728068 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.729271 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.731524 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.732152 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.732302 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.732744 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.733411 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.733567 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.733647 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.733725 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 04:59:04.750114 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.744622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:38649 (local address 127.20.169.67:34649)
0927 04:59:04.744893 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.744900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.745176 (+ 276us) server_negotiation.cc:407] Connection header received
0927 04:59:04.745404 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.745408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.745473 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.745593 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:04.746161 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.746876 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.748287 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.748719 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.748892 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.749298 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.749405 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.749542 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.749701 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.749779 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 04:59:04.750174 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.744434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34649 (local address 127.20.169.66:38649)
0927 04:59:04.745072 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.745091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.745252 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.745606 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.745610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.745622 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:04.745936 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.745945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.747019 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.747025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.748147 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.748159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.748917 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.748923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.749053 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.749102 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.749554 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.749572 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.749677 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.749816 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":481,"threads_started":1}
I20260927 04:59:04.799897 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.785082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:44013 (local address 127.20.169.66:35957)
0927 04:59:04.785284 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:04.785512 (+ 228us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:04.793560 (+ 8048us) server_negotiation.cc:407] Connection header received
0927 04:59:04.793842 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:04.793847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:04.793961 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:04.794044 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:04.794609 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.795606 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.797763 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.798525 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.798667 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.799097 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.799229 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.799355 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:04.799477 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 04:59:04.799556 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 04:59:04.799935 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:04.785133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:35957 (local address 127.20.169.67:44013)
0927 04:59:04.793413 (+ 8280us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:04.793469 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:04.793684 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:04.794064 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:04.794068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:04.794081 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:04.794408 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.794417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.795811 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.795816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.797617 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:04.797629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:04.798677 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:04.798682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:04.798846 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:04.798900 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:04.799369 (+ 469us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:04.799382 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:59:04.799458 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:59:04.799599 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8043,"thread_start_us":7869,"threads_started":1}
I20260927 04:59:04.838040 22928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:04.838353 22928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:04.842549 22928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Sending DeleteTablet for 3 replicas of tablet 5650992cfc9b45e1a2d8c67f92cbe204
I20260927 04:59:04.843721 23067 tablet_service.cc:1514] Processing DeleteTablet for tablet 5650992cfc9b45e1a2d8c67f92cbe204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:04 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53408
I20260927 04:59:04.843811 23207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5650992cfc9b45e1a2d8c67f92cbe204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:04 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260927 04:59:04.845723 23351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5650992cfc9b45e1a2d8c67f92cbe204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:04 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:36752
I20260927 04:59:04.853001 22929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:04.853817 22929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:04.856318 23489 tablet_replica.cc:330] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: stopping tablet replica
I20260927 04:59:04.856925 23489 raft_consensus.cc:2229] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:04.857317 23489 raft_consensus.cc:2258] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:04.859329 23489 ts_tablet_manager.cc:1905] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:04.862213 23067 tablet_service.cc:1467] Processing CreateTablet for tablet 68da53e4295d4a71a4412c5f13f48ea3 (DEFAULT_TABLE table=test-table [id=4a7030e35a8844cf9c5ca1eebd58a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:04.862619 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68da53e4295d4a71a4412c5f13f48ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.862960 23206 tablet_service.cc:1467] Processing CreateTablet for tablet 68da53e4295d4a71a4412c5f13f48ea3 (DEFAULT_TABLE table=test-table [id=4a7030e35a8844cf9c5ca1eebd58a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:04.863417 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68da53e4295d4a71a4412c5f13f48ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.866823 23350 tablet_service.cc:1467] Processing CreateTablet for tablet 68da53e4295d4a71a4412c5f13f48ea3 (DEFAULT_TABLE table=test-table [id=4a7030e35a8844cf9c5ca1eebd58a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:04.867370 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68da53e4295d4a71a4412c5f13f48ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:04.888273 23491 tablet_replica.cc:330] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: stopping tablet replica
I20260927 04:59:04.888785 23491 raft_consensus.cc:2229] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:04.889156 23491 raft_consensus.cc:2258] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:04.888299 23495 tablet_bootstrap.cc:492] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Bootstrap starting.
I20260927 04:59:04.889914 23494 tablet_bootstrap.cc:492] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Bootstrap starting.
I20260927 04:59:04.891990 23495 tablet_bootstrap.cc:654] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.893692 23494 tablet_bootstrap.cc:654] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.892637 23491 ts_tablet_manager.cc:1905] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:04.904290 23490 tablet_replica.cc:330] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: stopping tablet replica
I20260927 04:59:04.905036 23489 ts_tablet_manager.cc:1918] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:04.904745 23492 tablet_bootstrap.cc:492] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Bootstrap starting.
I20260927 04:59:04.905211 23489 log.cc:1196] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/5650992cfc9b45e1a2d8c67f92cbe204
I20260927 04:59:04.905758 23489 ts_tablet_manager.cc:1939] T 5650992cfc9b45e1a2d8c67f92cbe204 P 59d88268997847f8a5cc96f0421e77b3: Deleting consensus metadata
I20260927 04:59:04.907717 23492 tablet_bootstrap.cc:654] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:04.907917 23490 raft_consensus.cc:2229] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:04.909518 23490 raft_consensus.cc:2258] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:04.911815 23490 ts_tablet_manager.cc:1905] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:04.912209 23494 tablet_bootstrap.cc:492] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: No bootstrap required, opened a new log
I20260927 04:59:04.912364 23494 ts_tablet_manager.cc:1397] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20260927 04:59:04.913241 23494 raft_consensus.cc:348] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.913496 23494 raft_consensus.cc:374] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.913586 23494 raft_consensus.cc:729] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d88268997847f8a5cc96f0421e77b3, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.914258 23494 consensus_queue.cc:260] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.915563 23495 tablet_bootstrap.cc:492] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: No bootstrap required, opened a new log
I20260927 04:59:04.915725 23495 ts_tablet_manager.cc:1397] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20260927 04:59:04.916536 23495 raft_consensus.cc:348] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.916786 23495 raft_consensus.cc:374] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.916913 23495 raft_consensus.cc:729] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.917596 23492 tablet_bootstrap.cc:492] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: No bootstrap required, opened a new log
I20260927 04:59:04.917225 23495 consensus_queue.cc:260] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.917743 23492 ts_tablet_manager.cc:1397] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20260927 04:59:04.919435 23492 raft_consensus.cc:348] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.919648 23492 raft_consensus.cc:374] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:04.919732 23492 raft_consensus.cc:729] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Initialized, Role: FOLLOWER
I20260927 04:59:04.919993 23492 consensus_queue.cc:260] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:04.920416 23495 ts_tablet_manager.cc:1428] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260927 04:59:04.921986 23492 ts_tablet_manager.cc:1428] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 04:59:04.922489 23494 ts_tablet_manager.cc:1428] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260927 04:59:04.987152 22913 catalog_manager.cc:4841] TS 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679): tablet 5650992cfc9b45e1a2d8c67f92cbe204 (table test-table [id=76e1242d6c92436db8b7b754cff40cfd]) successfully deleted
I20260927 04:59:05.010169 23491 ts_tablet_manager.cc:1918] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:05.010452 23491 log.cc:1196] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/5650992cfc9b45e1a2d8c67f92cbe204
I20260927 04:59:05.011119 23491 ts_tablet_manager.cc:1939] T 5650992cfc9b45e1a2d8c67f92cbe204 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting consensus metadata
I20260927 04:59:05.076530 22915 catalog_manager.cc:4841] TS 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649): tablet 5650992cfc9b45e1a2d8c67f92cbe204 (table test-table [id=76e1242d6c92436db8b7b754cff40cfd]) successfully deleted
I20260927 04:59:05.107539 23490 ts_tablet_manager.cc:1918] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:05.107782 23490 log.cc:1196] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/5650992cfc9b45e1a2d8c67f92cbe204
I20260927 04:59:05.108402 23490 ts_tablet_manager.cc:1939] T 5650992cfc9b45e1a2d8c67f92cbe204 P f0012c42fac845918cc8629b01cac45f: Deleting consensus metadata
I20260927 04:59:05.110533 22915 catalog_manager.cc:4841] TS f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957): tablet 5650992cfc9b45e1a2d8c67f92cbe204 (table test-table [id=76e1242d6c92436db8b7b754cff40cfd]) successfully deleted
I20260927 04:59:05.314481 23448 raft_consensus.cc:482] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:05.314760 23448 raft_consensus.cc:504] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:05.315687 23448 leader_election.cc:288] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
I20260927 04:59:05.316249 23086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da53e4295d4a71a4412c5f13f48ea3" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3" is_pre_election: true
I20260927 04:59:05.316344 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da53e4295d4a71a4412c5f13f48ea3" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" is_pre_election: true
I20260927 04:59:05.316551 23086 raft_consensus.cc:2454] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 0.
I20260927 04:59:05.316566 23371 raft_consensus.cc:2454] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 0.
I20260927 04:59:05.317047 23160 leader_election.cc:302] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:05.317414 23448 raft_consensus.cc:2781] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:05.317552 23448 raft_consensus.cc:482] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:05.317706 23448 raft_consensus.cc:3037] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:05.408200 23448 raft_consensus.cc:504] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:05.410166 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da53e4295d4a71a4412c5f13f48ea3" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7"
I20260927 04:59:05.410439 23371 raft_consensus.cc:3037] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:05.414112 23086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da53e4295d4a71a4412c5f13f48ea3" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3"
I20260927 04:59:05.414389 23086 raft_consensus.cc:3037] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:05.417027 23448 leader_election.cc:288] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 election: Requested vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
I20260927 04:59:05.514206 23371 raft_consensus.cc:2454] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 1.
I20260927 04:59:05.514220 23086 raft_consensus.cc:2454] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 1.
I20260927 04:59:05.514959 23160 leader_election.cc:302] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:05.515445 23448 raft_consensus.cc:2781] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:05.516695 23448 raft_consensus.cc:686] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 1 LEADER]: Becoming Leader. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Running, Role: LEADER
I20260927 04:59:05.517087 23448 consensus_queue.cc:237] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:05.520756 22929 catalog_manager.cc:5495] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f reported cstate change: term changed from 0 to 1, leader changed from <none> to f0012c42fac845918cc8629b01cac45f (127.20.169.66). New cstate: current_term: 1 leader_uuid: "f0012c42fac845918cc8629b01cac45f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } health_report { overall_health: UNKNOWN } } }
I20260927 04:59:05.594652 23371 raft_consensus.cc:1261] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Refusing update from remote peer f0012c42fac845918cc8629b01cac45f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:05.594414 23086 raft_consensus.cc:1261] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Refusing update from remote peer f0012c42fac845918cc8629b01cac45f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:05.595428 23499 consensus_queue.cc:1035] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:05.595917 23448 consensus_queue.cc:1035] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 04:59:05.619993 23506 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 04:59:07.217639 22929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:07.217929 22929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:07.223059 22929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Sending DeleteTablet for 3 replicas of tablet 68da53e4295d4a71a4412c5f13f48ea3
I20260927 04:59:07.224036 23206 tablet_service.cc:1514] Processing DeleteTablet for tablet 68da53e4295d4a71a4412c5f13f48ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:07 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260927 04:59:07.224081 23067 tablet_service.cc:1514] Processing DeleteTablet for tablet 68da53e4295d4a71a4412c5f13f48ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:07 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53408
I20260927 04:59:07.224175 23351 tablet_service.cc:1514] Processing DeleteTablet for tablet 68da53e4295d4a71a4412c5f13f48ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:07 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:36752
I20260927 04:59:07.236979 23541 tablet_replica.cc:330] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: stopping tablet replica
I20260927 04:59:07.237341 23541 raft_consensus.cc:2229] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:07.237803 23541 raft_consensus.cc:2258] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:07.238480 23539 tablet_replica.cc:330] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: stopping tablet replica
I20260927 04:59:07.238765 23539 raft_consensus.cc:2229] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:07.239257 23541 ts_tablet_manager.cc:1905] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:07.239331 23539 raft_consensus.cc:2258] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:07.240437 23539 ts_tablet_manager.cc:1905] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:07.240631 23540 tablet_replica.cc:330] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: stopping tablet replica
I20260927 04:59:07.240937 23540 raft_consensus.cc:2229] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:07.241305 23540 raft_consensus.cc:2258] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:07.242357 23540 ts_tablet_manager.cc:1905] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:07.257711 23539 ts_tablet_manager.cc:1918] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:59:07.257892 23539 log.cc:1196] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/68da53e4295d4a71a4412c5f13f48ea3
I20260927 04:59:07.258005 23541 ts_tablet_manager.cc:1918] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:59:07.258148 23541 log.cc:1196] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/68da53e4295d4a71a4412c5f13f48ea3
I20260927 04:59:07.258705 23541 ts_tablet_manager.cc:1939] T 68da53e4295d4a71a4412c5f13f48ea3 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting consensus metadata
I20260927 04:59:07.259003 23539 ts_tablet_manager.cc:1939] T 68da53e4295d4a71a4412c5f13f48ea3 P f0012c42fac845918cc8629b01cac45f: Deleting consensus metadata
Sep 27 04:59:07 dist-test-slave-39f2 krb5kdc[22884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260927 04:59:07.327126 22915 catalog_manager.cc:4841] TS 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649): tablet 68da53e4295d4a71a4412c5f13f48ea3 (table test-table [id=4a7030e35a8844cf9c5ca1eebd58a6c4]) successfully deleted
I20260927 04:59:07.327365 22915 catalog_manager.cc:4841] TS f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957): tablet 68da53e4295d4a71a4412c5f13f48ea3 (table test-table [id=4a7030e35a8844cf9c5ca1eebd58a6c4]) successfully deleted
W20260927 04:59:07.357327 21157 mini_kdc.cc:317] Time spent kinit for test-user: real 0.132s user 0.004s sys 0.009s
I20260927 04:59:07.357542 23540 ts_tablet_manager.cc:1918] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:59:07.357725 23540 log.cc:1196] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/68da53e4295d4a71a4412c5f13f48ea3
I20260927 04:59:07.358191 23540 ts_tablet_manager.cc:1939] T 68da53e4295d4a71a4412c5f13f48ea3 P 59d88268997847f8a5cc96f0421e77b3: Deleting consensus metadata
I20260927 04:59:07.360216 22913 catalog_manager.cc:4841] TS 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679): tablet 68da53e4295d4a71a4412c5f13f48ea3 (table test-table [id=4a7030e35a8844cf9c5ca1eebd58a6c4]) successfully deleted
Sep 27 04:59:07 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:07.377358 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:07.361616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.20.169.126:38515)
0927 04:59:07.362449 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:07.362457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:07.362907 (+ 450us) server_negotiation.cc:407] Connection header received
0927 04:59:07.363154 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:07.363158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:07.363225 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:07.363404 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:07.365264 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.366050 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.367315 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.367720 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.367868 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:07.371827 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:07.371832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:07.371835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:07.371867 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:07.375070 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:07.375672 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:07.375675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:07.375678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:07.375797 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:07.376098 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:07.376101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:07.376103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:07.376327 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:07.376510 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:07.376759 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 04:59:07.376968 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":364,"threads_started":1}
I20260927 04:59:07.381394 22929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:07.382108 22929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:07.388932 23067 tablet_service.cc:1467] Processing CreateTablet for tablet 8808dbf26a264f768fa6edbb0e80625c (DEFAULT_TABLE table=test-table [id=56f4ff9be97f46d3b2be094b2fecdc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:07.389202 23351 tablet_service.cc:1467] Processing CreateTablet for tablet 8808dbf26a264f768fa6edbb0e80625c (DEFAULT_TABLE table=test-table [id=56f4ff9be97f46d3b2be094b2fecdc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:07.389467 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8808dbf26a264f768fa6edbb0e80625c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:07.389654 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8808dbf26a264f768fa6edbb0e80625c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:07.394305 23206 tablet_service.cc:1467] Processing CreateTablet for tablet 8808dbf26a264f768fa6edbb0e80625c (DEFAULT_TABLE table=test-table [id=56f4ff9be97f46d3b2be094b2fecdc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:07.394778 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8808dbf26a264f768fa6edbb0e80625c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:07.403447 23551 tablet_bootstrap.cc:492] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Bootstrap starting.
I20260927 04:59:07.404328 23550 tablet_bootstrap.cc:492] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Bootstrap starting.
I20260927 04:59:07.405213 23552 tablet_bootstrap.cc:492] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Bootstrap starting.
I20260927 04:59:07.407588 23550 tablet_bootstrap.cc:654] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:07.407660 23551 tablet_bootstrap.cc:654] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:07.409376 23552 tablet_bootstrap.cc:654] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:07.413528 23552 tablet_bootstrap.cc:492] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: No bootstrap required, opened a new log
I20260927 04:59:07.413748 23552 ts_tablet_manager.cc:1397] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 04:59:07.413947 23550 tablet_bootstrap.cc:492] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: No bootstrap required, opened a new log
I20260927 04:59:07.414149 23550 ts_tablet_manager.cc:1397] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 04:59:07.414819 23551 tablet_bootstrap.cc:492] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: No bootstrap required, opened a new log
I20260927 04:59:07.415081 23551 ts_tablet_manager.cc:1397] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260927 04:59:07.415185 23550 raft_consensus.cc:348] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.415504 23550 raft_consensus.cc:374] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:07.415371 23552 raft_consensus.cc:348] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.415711 23550 raft_consensus.cc:729] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Initialized, Role: FOLLOWER
I20260927 04:59:07.415799 23552 raft_consensus.cc:374] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:07.415975 23552 raft_consensus.cc:729] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Initialized, Role: FOLLOWER
I20260927 04:59:07.416087 23550 consensus_queue.cc:260] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.416332 23552 consensus_queue.cc:260] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.416833 23550 ts_tablet_manager.cc:1428] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:59:07.416525 23551 raft_consensus.cc:348] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.416975 23551 raft_consensus.cc:374] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:07.417200 23551 raft_consensus.cc:729] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d88268997847f8a5cc96f0421e77b3, State: Initialized, Role: FOLLOWER
I20260927 04:59:07.417284 23552 ts_tablet_manager.cc:1428] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:07.418212 23551 consensus_queue.cc:260] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.421816 23551 ts_tablet_manager.cc:1428] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260927 04:59:07.756361 23534 raft_consensus.cc:482] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:07.756551 23534 raft_consensus.cc:504] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.757380 23534 leader_election.cc:288] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957)
I20260927 04:59:07.758033 23086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808dbf26a264f768fa6edbb0e80625c" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3" is_pre_election: true
I20260927 04:59:07.758034 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808dbf26a264f768fa6edbb0e80625c" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0012c42fac845918cc8629b01cac45f" is_pre_election: true
I20260927 04:59:07.758314 23086 raft_consensus.cc:2454] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 0.
I20260927 04:59:07.758291 23227 raft_consensus.cc:2454] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 0.
I20260927 04:59:07.758846 23299 leader_election.cc:302] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, 59d88268997847f8a5cc96f0421e77b3, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:07.759136 23534 raft_consensus.cc:2781] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:07.759260 23534 raft_consensus.cc:482] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:07.759426 23534 raft_consensus.cc:3037] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:07.762864 23534 raft_consensus.cc:504] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.763685 23534 leader_election.cc:288] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 election: Requested vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957)
I20260927 04:59:07.764115 23086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808dbf26a264f768fa6edbb0e80625c" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3"
I20260927 04:59:07.764360 23086 raft_consensus.cc:3037] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:07.764391 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808dbf26a264f768fa6edbb0e80625c" candidate_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0012c42fac845918cc8629b01cac45f"
I20260927 04:59:07.764673 23227 raft_consensus.cc:3037] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:07.766872 23086 raft_consensus.cc:2454] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 1.
I20260927 04:59:07.767328 23297 leader_election.cc:302] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, 59d88268997847f8a5cc96f0421e77b3; no voters:
I20260927 04:59:07.767594 23534 raft_consensus.cc:2781] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:07.768452 23534 raft_consensus.cc:686] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 LEADER]: Becoming Leader. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Running, Role: LEADER
I20260927 04:59:07.768716 23227 raft_consensus.cc:2454] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934a2395abf4a5ab41b6c9bb53bcab7 in term 1.
I20260927 04:59:07.768743 23534 consensus_queue.cc:237] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:07.772172 22928 catalog_manager.cc:5495] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } health_report { overall_health: HEALTHY } } }
I20260927 04:59:07.901796 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:07.890212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.20.169.67:34649)
0927 04:59:07.894336 (+ 4124us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:07.894343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:07.894369 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:59:07.894432 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:07.894438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:07.894593 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:07.894691 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:07.895565 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.897151 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.899128 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.899939 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.900084 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:07.900547 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:07.900705 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:07.900791 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:07.901273 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 04:59:07.901388 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3982,"thread_start_us":3838,"threads_started":1}
I20260927 04:59:07.917759 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:07.912000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:41991 (local address 127.20.169.67:34649)
0927 04:59:07.912233 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:07.912251 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:07.912627 (+ 376us) server_negotiation.cc:407] Connection header received
0927 04:59:07.912757 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:07.912761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:07.912890 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:07.912995 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:07.913626 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.914571 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.915895 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.916362 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.916506 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:07.916888 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:07.916999 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:07.917116 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:07.917344 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 04:59:07.917421 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":57}
I20260927 04:59:07.917807 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:07.911910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34649 (local address 127.20.169.67:41991)
0927 04:59:07.912533 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:07.912548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:07.912657 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:07.913005 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:07.913010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:07.913020 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:07.913481 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:07.913491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.914697 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.914703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:07.915750 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:07.915761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:07.916524 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:07.916530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:07.916652 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:07.916685 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:07.917136 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:07.917176 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 04:59:07.917326 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 04:59:07.917475 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":49,"thread_start_us":332,"threads_started":1}
I20260927 04:59:07.925940 23086 raft_consensus.cc:1261] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Refusing update from remote peer 3934a2395abf4a5ab41b6c9bb53bcab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:07.926554 23557 consensus_queue.cc:1035] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:07.941695 23227 raft_consensus.cc:1261] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Refusing update from remote peer 3934a2395abf4a5ab41b6c9bb53bcab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:07.942297 23534 consensus_queue.cc:1035] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:07.946476 23560 mvcc.cc:205] Tried to move back new op lower bound from 7333827165934313472 to 7333827165263695872. Current Snapshot: MvccSnapshot[applied={T|T < 7333827165934313472}]
I20260927 04:59:07.988564 22928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:07.989087 22928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:07.993057 22928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Sending DeleteTablet for 3 replicas of tablet 8808dbf26a264f768fa6edbb0e80625c
I20260927 04:59:07.994362 23067 tablet_service.cc:1514] Processing DeleteTablet for tablet 8808dbf26a264f768fa6edbb0e80625c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:07 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53408
I20260927 04:59:07.994657 23206 tablet_service.cc:1514] Processing DeleteTablet for tablet 8808dbf26a264f768fa6edbb0e80625c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:07 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260927 04:59:07.995064 23351 tablet_service.cc:1514] Processing DeleteTablet for tablet 8808dbf26a264f768fa6edbb0e80625c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:07 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:36752
I20260927 04:59:07.995842 23580 tablet_replica.cc:330] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: stopping tablet replica
I20260927 04:59:07.995736 22928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 04:59:07.996534 23580 raft_consensus.cc:2229] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 04:59:07.996762 22928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:07.997274 23580 raft_consensus.cc:2258] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:08.000249 23580 ts_tablet_manager.cc:1905] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:08.010433 23067 tablet_service.cc:1467] Processing CreateTablet for tablet cd9f6b58d0b84fe389b64e4905e3c7a7 (DEFAULT_TABLE table=test-table [id=b74d9d619e7f4fb99d9f2f6e4ba787ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:08.011204 23582 tablet_replica.cc:330] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: stopping tablet replica
I20260927 04:59:08.011819 23582 raft_consensus.cc:2229] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:08.013010 23351 tablet_service.cc:1467] Processing CreateTablet for tablet cd9f6b58d0b84fe389b64e4905e3c7a7 (DEFAULT_TABLE table=test-table [id=b74d9d619e7f4fb99d9f2f6e4ba787ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:08.014459 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9f6b58d0b84fe389b64e4905e3c7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:08.014483 23207 tablet_service.cc:1467] Processing CreateTablet for tablet cd9f6b58d0b84fe389b64e4905e3c7a7 (DEFAULT_TABLE table=test-table [id=b74d9d619e7f4fb99d9f2f6e4ba787ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:08.014693 23581 tablet_replica.cc:330] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: stopping tablet replica
I20260927 04:59:08.015084 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9f6b58d0b84fe389b64e4905e3c7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:08.015725 23582 raft_consensus.cc:2258] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:08.015116 23581 raft_consensus.cc:2229] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:08.017303 23581 raft_consensus.cc:2258] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:08.018460 23582 ts_tablet_manager.cc:1905] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:08.018668 23581 ts_tablet_manager.cc:1905] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:08.025853 23583 tablet_bootstrap.cc:492] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Bootstrap starting.
I20260927 04:59:08.029098 23583 tablet_bootstrap.cc:654] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:08.033773 23585 tablet_bootstrap.cc:492] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Bootstrap starting.
I20260927 04:59:08.035295 23583 tablet_bootstrap.cc:492] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: No bootstrap required, opened a new log
I20260927 04:59:08.035537 23583 ts_tablet_manager.cc:1397] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260927 04:59:08.036754 23583 raft_consensus.cc:348] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.037236 23583 raft_consensus.cc:374] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:08.037433 23583 raft_consensus.cc:729] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Initialized, Role: FOLLOWER
I20260927 04:59:08.038021 23585 tablet_bootstrap.cc:654] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:08.037875 23583 consensus_queue.cc:260] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.038684 23583 ts_tablet_manager.cc:1428] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:08.042152 23585 tablet_bootstrap.cc:492] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: No bootstrap required, opened a new log
I20260927 04:59:08.042412 23585 ts_tablet_manager.cc:1397] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 04:59:08.043815 23585 raft_consensus.cc:348] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.044369 23585 raft_consensus.cc:374] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:08.044862 23585 raft_consensus.cc:729] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934a2395abf4a5ab41b6c9bb53bcab7, State: Initialized, Role: FOLLOWER
I20260927 04:59:08.045310 23585 consensus_queue.cc:260] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.046097 23585 ts_tablet_manager.cc:1428] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:59:08.064483 23067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9f6b58d0b84fe389b64e4905e3c7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:08.068687 23581 ts_tablet_manager.cc:1918] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:08.068923 23581 log.cc:1196] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/8808dbf26a264f768fa6edbb0e80625c
I20260927 04:59:08.069711 23581 ts_tablet_manager.cc:1939] T 8808dbf26a264f768fa6edbb0e80625c P f0012c42fac845918cc8629b01cac45f: Deleting consensus metadata
I20260927 04:59:08.075912 23587 tablet_bootstrap.cc:492] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Bootstrap starting.
I20260927 04:59:08.079083 23587 tablet_bootstrap.cc:654] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:08.082790 23587 tablet_bootstrap.cc:492] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: No bootstrap required, opened a new log
I20260927 04:59:08.083025 23587 ts_tablet_manager.cc:1397] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260927 04:59:08.084239 23587 raft_consensus.cc:348] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.084635 23587 raft_consensus.cc:374] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:08.084856 23587 raft_consensus.cc:729] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d88268997847f8a5cc96f0421e77b3, State: Initialized, Role: FOLLOWER
I20260927 04:59:08.085265 23587 consensus_queue.cc:260] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.086225 23587 ts_tablet_manager.cc:1428] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 04:59:08.161131 22915 catalog_manager.cc:4841] TS f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957): tablet 8808dbf26a264f768fa6edbb0e80625c (table test-table [id=56f4ff9be97f46d3b2be094b2fecdc60]) successfully deleted
I20260927 04:59:08.162329 23574 raft_consensus.cc:482] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:08.162689 23574 raft_consensus.cc:504] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.163841 23574 leader_election.cc:288] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
I20260927 04:59:08.164499 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9f6b58d0b84fe389b64e4905e3c7a7" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" is_pre_election: true
I20260927 04:59:08.164499 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9f6b58d0b84fe389b64e4905e3c7a7" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3" is_pre_election: true
I20260927 04:59:08.164904 23087 raft_consensus.cc:2454] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 0.
I20260927 04:59:08.164940 23371 raft_consensus.cc:2454] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 0.
I20260927 04:59:08.165472 23158 leader_election.cc:302] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d88268997847f8a5cc96f0421e77b3, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:08.165874 23574 raft_consensus.cc:2781] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:08.166077 23574 raft_consensus.cc:482] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:08.166225 23574 raft_consensus.cc:3037] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:08.167306 23582 ts_tablet_manager.cc:1918] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:08.167384 23580 ts_tablet_manager.cc:1918] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:08.167528 23582 log.cc:1196] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/8808dbf26a264f768fa6edbb0e80625c
I20260927 04:59:08.167591 23580 log.cc:1196] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/8808dbf26a264f768fa6edbb0e80625c
I20260927 04:59:08.168213 23582 ts_tablet_manager.cc:1939] T 8808dbf26a264f768fa6edbb0e80625c P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting consensus metadata
I20260927 04:59:08.168269 23580 ts_tablet_manager.cc:1939] T 8808dbf26a264f768fa6edbb0e80625c P 59d88268997847f8a5cc96f0421e77b3: Deleting consensus metadata
I20260927 04:59:08.173097 23574 raft_consensus.cc:504] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.173408 22913 catalog_manager.cc:4841] TS 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679): tablet 8808dbf26a264f768fa6edbb0e80625c (table test-table [id=56f4ff9be97f46d3b2be094b2fecdc60]) successfully deleted
I20260927 04:59:08.174067 22915 catalog_manager.cc:4841] TS 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649): tablet 8808dbf26a264f768fa6edbb0e80625c (table test-table [id=56f4ff9be97f46d3b2be094b2fecdc60]) successfully deleted
I20260927 04:59:08.174450 23574 leader_election.cc:288] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 election: Requested vote from peers 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679), 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649)
I20260927 04:59:08.174463 23087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9f6b58d0b84fe389b64e4905e3c7a7" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d88268997847f8a5cc96f0421e77b3"
I20260927 04:59:08.174737 23087 raft_consensus.cc:3037] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:08.175105 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9f6b58d0b84fe389b64e4905e3c7a7" candidate_uuid: "f0012c42fac845918cc8629b01cac45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7"
I20260927 04:59:08.175457 23371 raft_consensus.cc:3037] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:08.273679 23087 raft_consensus.cc:2454] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 1.
I20260927 04:59:08.274415 23371 raft_consensus.cc:2454] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0012c42fac845918cc8629b01cac45f in term 1.
I20260927 04:59:08.274983 23160 leader_election.cc:302] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934a2395abf4a5ab41b6c9bb53bcab7, f0012c42fac845918cc8629b01cac45f; no voters:
I20260927 04:59:08.275496 23574 raft_consensus.cc:2781] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:08.279181 23574 raft_consensus.cc:686] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 1 LEADER]: Becoming Leader. State: Replica: f0012c42fac845918cc8629b01cac45f, State: Running, Role: LEADER
I20260927 04:59:08.279631 23574 consensus_queue.cc:237] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } }
I20260927 04:59:08.282876 22928 catalog_manager.cc:5495] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f reported cstate change: term changed from 0 to 1, leader changed from <none> to f0012c42fac845918cc8629b01cac45f (127.20.169.66). New cstate: current_term: 1 leader_uuid: "f0012c42fac845918cc8629b01cac45f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0012c42fac845918cc8629b01cac45f" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 } health_report { overall_health: UNKNOWN } } }
I20260927 04:59:08.342437 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:08.332526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.20.169.66:35957)
0927 04:59:08.337775 (+ 5249us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:08.337796 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:08.337821 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:59:08.337904 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:08.337911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:08.338011 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:08.338137 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:08.338552 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:08.339329 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:08.340637 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:08.341133 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:08.341267 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:08.341702 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:08.341807 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:08.341907 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:08.342024 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 04:59:08.342097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5092,"thread_start_us":4880,"threads_started":1}
I20260927 04:59:08.352639 23086 raft_consensus.cc:1261] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Refusing update from remote peer f0012c42fac845918cc8629b01cac45f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:08.353371 23574 consensus_queue.cc:1035] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d88268997847f8a5cc96f0421e77b3" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:08.353888 23371 raft_consensus.cc:1261] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Refusing update from remote peer f0012c42fac845918cc8629b01cac45f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:08.354534 23574 consensus_queue.cc:1035] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934a2395abf4a5ab41b6c9bb53bcab7" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 04:59:08.386927 23560 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 04:59:09.994180 22927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:09.994529 22927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:10.000715 22927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bcaf291e1e34ed89410b76ee13fd63e: Sending DeleteTablet for 3 replicas of tablet cd9f6b58d0b84fe389b64e4905e3c7a7
I20260927 04:59:10.001701 23207 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9f6b58d0b84fe389b64e4905e3c7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:09 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260927 04:59:10.001703 23067 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9f6b58d0b84fe389b64e4905e3c7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:09 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53408
I20260927 04:59:10.001792 23351 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9f6b58d0b84fe389b64e4905e3c7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:09 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:36752
I20260927 04:59:10.002770 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23001
I20260927 04:59:10.007956 23623 tablet_replica.cc:330] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: stopping tablet replica
I20260927 04:59:10.008303 23623 raft_consensus.cc:2229] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:10.008848 23623 raft_consensus.cc:2258] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:10.009048 23622 tablet_replica.cc:330] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: stopping tablet replica
I20260927 04:59:10.009680 23622 raft_consensus.cc:2229] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:10.009966 23622 raft_consensus.cc:2258] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:10.010051 23623 ts_tablet_manager.cc:1905] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:10.011211 23622 ts_tablet_manager.cc:1905] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:10.017071 23621 tablet_replica.cc:330] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: stopping tablet replica
I20260927 04:59:10.017567 23621 raft_consensus.cc:2229] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:10.018178 23621 raft_consensus.cc:2258] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:10.019490 23621 ts_tablet_manager.cc:1905] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:10.019724 23623 ts_tablet_manager.cc:1918] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:59:10.019958 23622 ts_tablet_manager.cc:1918] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:59:10.020038 23623 log.cc:1196] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/cd9f6b58d0b84fe389b64e4905e3c7a7
I20260927 04:59:10.020274 23622 log.cc:1196] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/cd9f6b58d0b84fe389b64e4905e3c7a7
I20260927 04:59:10.020715 23623 ts_tablet_manager.cc:1939] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 59d88268997847f8a5cc96f0421e77b3: Deleting consensus metadata
I20260927 04:59:10.020918 23622 ts_tablet_manager.cc:1939] T cd9f6b58d0b84fe389b64e4905e3c7a7 P 3934a2395abf4a5ab41b6c9bb53bcab7: Deleting consensus metadata
I20260927 04:59:10.022022 22913 catalog_manager.cc:4841] TS 59d88268997847f8a5cc96f0421e77b3 (127.20.169.65:41679): tablet cd9f6b58d0b84fe389b64e4905e3c7a7 (table test-table [id=b74d9d619e7f4fb99d9f2f6e4ba787ba]) successfully deleted
I20260927 04:59:10.022022 22915 catalog_manager.cc:4841] TS 3934a2395abf4a5ab41b6c9bb53bcab7 (127.20.169.67:34649): tablet cd9f6b58d0b84fe389b64e4905e3c7a7 (table test-table [id=b74d9d619e7f4fb99d9f2f6e4ba787ba]) successfully deleted
I20260927 04:59:10.025748 23621 ts_tablet_manager.cc:1918] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:59:10.025862 23621 log.cc:1196] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/cd9f6b58d0b84fe389b64e4905e3c7a7
I20260927 04:59:10.026342 23621 ts_tablet_manager.cc:1939] T cd9f6b58d0b84fe389b64e4905e3c7a7 P f0012c42fac845918cc8629b01cac45f: Deleting consensus metadata
I20260927 04:59:10.027400 22915 catalog_manager.cc:4841] TS f0012c42fac845918cc8629b01cac45f (127.20.169.66:35957): tablet cd9f6b58d0b84fe389b64e4905e3c7a7 (table test-table [id=b74d9d619e7f4fb99d9f2f6e4ba787ba]) successfully deleted
Sep 27 04:59:10 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:10.036677 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:10.008465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.20.169.65:41679)
0927 04:59:10.010500 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:10.010509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:10.013277 (+ 2768us) server_negotiation.cc:407] Connection header received
0927 04:59:10.013653 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:10.013662 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:10.013783 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:10.013954 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:10.016975 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:10.018766 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:10.021040 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:10.022130 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:10.022245 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:10.030987 (+ 8742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:10.031004 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:10.031022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:10.031064 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:10.033895 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:10.034607 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:10.034616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:10.034619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:10.034739 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:10.035155 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:10.035159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:10.035169 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:10.035518 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:10.035837 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:10.036116 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 04:59:10.036338 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":2063,"threads_started":1}
W20260927 04:59:10.037331 23132 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53412
W20260927 04:59:10.037955 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:10.038162 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 23001
I20260927 04:59:10.056988 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23142
Sep 27 04:59:10 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:10.071880 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:10.057737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.20.169.66:35957)
0927 04:59:10.058270 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:10.058274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:10.058291 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:10.058435 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:10.058438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:10.058526 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:10.058617 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:10.059891 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:10.060616 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:10.061966 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:10.062379 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:10.062483 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:10.066461 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:10.066476 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:10.066493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:10.066528 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:10.069254 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:10.069957 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:10.069966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:10.069968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:10.070125 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:10.070513 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:10.070516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:10.070518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:10.070876 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:10.071113 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:10.071431 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 04:59:10.071610 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
W20260927 04:59:10.072702 23272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39528
W20260927 04:59:10.073570 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:10.073720 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 23142
I20260927 04:59:10.097280 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23281
Sep 27 04:59:10 dist-test-slave-39f2 krb5kdc[22884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:10.112618 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:10.098104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.20.169.67:34649)
0927 04:59:10.098768 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:10.098772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:10.098790 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:59:10.098850 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:10.098853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:10.098921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:10.099019 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:10.100238 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:10.101130 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:10.102281 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:10.102668 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:10.102810 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:10.107253 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:10.107266 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:10.107282 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:10.107355 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:10.110287 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:10.110854 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:10.110862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:10.110864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:10.110999 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:10.111389 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:10.111393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:10.111395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:10.111717 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:10.111968 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:10.112191 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 04:59:10.112342 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":366,"threads_started":1}
W20260927 04:59:10.113306 23413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36784
W20260927 04:59:10.113982 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23281: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:10.114118 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 23281
I20260927 04:59:10.140542 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 22897
I20260927 04:59:10.141806 22969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:10.256922 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 22897
2026-09-27T04:59:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9706 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:10 dist-test-slave-39f2 krb5kdc[23631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:10 dist-test-slave-39f2 krb5kdc[23631](info): set up 2 sockets
Sep 27 04:59:10 dist-test-slave-39f2 krb5kdc[23631](info): commencing operation
krb5kdc: starting...
W20260927 04:59:11.357354 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:11 dist-test-slave-39f2 krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:11.557004 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:42941
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:42941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:11.893620 23644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:11.893879 23644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:11.894035 23644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:11.901218 23644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:11.901386 23644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:11.901489 23644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:11.901587 23644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:11.901657 23644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:11.911336 23644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:42941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:42941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:11.912734 23644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:11.914062 23644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:11.926136 23649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:11.926314 23652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:11.926827 23644 server_base.cc:1034] running on GCE node
W20260927 04:59:11.926905 23650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:11.927685 23644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:11.929101 23644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:11.930334 23644 hybrid_clock.cc:648] HybridClock initialized: now 1790485151930277 us; error 51 us; skew 500 ppm
I20260927 04:59:11.930671 23644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:11 dist-test-slave-39f2 krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485151, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:11.933874 23644 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:11.935483 23644 webserver.cc:460] Webserver started at http://127.20.169.126:44401/ using document root <none> and password file <none>
I20260927 04:59:11.936159 23644 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:11.936306 23644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:11.936616 23644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:11.939610 23644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "dbd97a6839b143fcaf6a5560609f9d6e"
format_stamp: "Formatted at 2026-09-27 04:59:11 on dist-test-slave-39f2"
server_key: "11e64b9dbf9368685ffec3100126ffbc"
server_key_iv: "308036f9ddb7e3c7e984a727f9563449"
server_key_version: "encryptionkey@0"
I20260927 04:59:11.940340 23644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "dbd97a6839b143fcaf6a5560609f9d6e"
format_stamp: "Formatted at 2026-09-27 04:59:11 on dist-test-slave-39f2"
server_key: "11e64b9dbf9368685ffec3100126ffbc"
server_key_iv: "308036f9ddb7e3c7e984a727f9563449"
server_key_version: "encryptionkey@0"
I20260927 04:59:11.945360 23644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 04:59:11.948617 23659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:11.950187 23644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:11.950413 23644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "dbd97a6839b143fcaf6a5560609f9d6e"
format_stamp: "Formatted at 2026-09-27 04:59:11 on dist-test-slave-39f2"
server_key: "11e64b9dbf9368685ffec3100126ffbc"
server_key_iv: "308036f9ddb7e3c7e984a727f9563449"
server_key_version: "encryptionkey@0"
I20260927 04:59:11.950662 23644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:11.971370 23644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:11.975674 23644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:11.975998 23644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:11.996362 23644 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:42941
I20260927 04:59:11.996405 23710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:42941 every 8 connection(s)
I20260927 04:59:11.997841 23644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:11.999193 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 23644
I20260927 04:59:11.999413 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:12.000264 21157 external_mini_cluster.cc:1428] Setting key 3bcc61b795b9424275d4e93a2b0cd596
I20260927 04:59:12.002876 23711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 04:59:12 dist-test-slave-39f2 krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:12.016336 23711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e: Bootstrap starting.
I20260927 04:59:12.020443 23711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:12.021631 23711 log.cc:824] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:12.025094 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:12.002397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.20.169.126:42941)
0927 04:59:12.003251 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:12.003262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:12.003312 (+ 50us) server_negotiation.cc:407] Connection header received
0927 04:59:12.004996 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:12.005018 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:12.005421 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:12.005766 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:12.007588 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:12.009076 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:12.010863 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:12.011583 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:12.011713 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:12.016344 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:12.016354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:12.016373 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:12.016427 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:12.020409 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:12.021201 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:12.021208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:12.021217 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:12.021431 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:12.021817 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:12.021822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:12.021825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:12.022482 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:12.022687 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:12.023323 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 04:59:12.023732 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":654,"thread_start_us":377,"threads_started":1}
I20260927 04:59:12.025202 23711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e: No bootstrap required, opened a new log
I20260927 04:59:12.032899 23711 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42941 } }
I20260927 04:59:12.033316 23711 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:12.033454 23711 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd97a6839b143fcaf6a5560609f9d6e, State: Initialized, Role: FOLLOWER
I20260927 04:59:12.034049 23711 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42941 } }
I20260927 04:59:12.034260 23711 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:12.034397 23711 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:12.034574 23711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:12.036612 23711 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42941 } }
I20260927 04:59:12.037192 23711 leader_election.cc:302] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd97a6839b143fcaf6a5560609f9d6e; no voters:
I20260927 04:59:12.037926 23711 leader_election.cc:288] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:12.038091 23716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:12.038769 23716 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [term 1 LEADER]: Becoming Leader. State: Replica: dbd97a6839b143fcaf6a5560609f9d6e, State: Running, Role: LEADER
I20260927 04:59:12.039319 23716 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42941 } }
I20260927 04:59:12.040022 23711 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:12.042292 23718 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd97a6839b143fcaf6a5560609f9d6e. Latest consensus state: current_term: 1 leader_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42941 } } }
I20260927 04:59:12.042250 23717 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd97a6839b143fcaf6a5560609f9d6e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42941 } } }
I20260927 04:59:12.042706 23718 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:12.042726 23717 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:12.044898 23723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:12.050602 23723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:12.059826 23723 catalog_manager.cc:1347] Generated new cluster ID: 63e5bf3a733543959172c14ae1f77f44
I20260927 04:59:12.059955 23723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:12.074404 23723 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:12.075632 23723 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:12.086990 23723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd97a6839b143fcaf6a5560609f9d6e: Generated new TSK 0
I20260927 04:59:12.087985 23723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:12.149087 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42941
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:12.485173 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:12.485513 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:12.485750 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:12.493104 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:12.493253 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:12.493450 23738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:12.503530 23738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:12.504829 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:12.506160 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:12.519381 23743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:12.519461 23744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:12.519461 23746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:12.520195 23738 server_base.cc:1034] running on GCE node
I20260927 04:59:12.520900 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:12.521872 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:12.523111 23738 hybrid_clock.cc:648] HybridClock initialized: now 1790485152523071 us; error 37 us; skew 500 ppm
I20260927 04:59:12.523521 23738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:12 dist-test-slave-39f2 krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485152, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:12.527025 23738 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:12.528614 23738 webserver.cc:460] Webserver started at http://127.20.169.65:45369/ using document root <none> and password file <none>
I20260927 04:59:12.529371 23738 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:12.529469 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:12.529722 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:12.532521 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "e1b5129042264ccaa273b451a2b00019"
format_stamp: "Formatted at 2026-09-27 04:59:12 on dist-test-slave-39f2"
server_key: "d1e910813a277625aeac48eb13931612"
server_key_iv: "340c9f09895f48de7cd6af93387a85dd"
server_key_version: "encryptionkey@0"
I20260927 04:59:12.533313 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "e1b5129042264ccaa273b451a2b00019"
format_stamp: "Formatted at 2026-09-27 04:59:12 on dist-test-slave-39f2"
server_key: "d1e910813a277625aeac48eb13931612"
server_key_iv: "340c9f09895f48de7cd6af93387a85dd"
server_key_version: "encryptionkey@0"
I20260927 04:59:12.538652 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:59:12.542304 23753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:12.543706 23738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:12.543906 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "e1b5129042264ccaa273b451a2b00019"
format_stamp: "Formatted at 2026-09-27 04:59:12 on dist-test-slave-39f2"
server_key: "d1e910813a277625aeac48eb13931612"
server_key_iv: "340c9f09895f48de7cd6af93387a85dd"
server_key_version: "encryptionkey@0"
I20260927 04:59:12.544116 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:12.558736 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:12.563239 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:12.563578 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:12.564625 23738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:12.566112 23738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:12.566227 23738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:12.566390 23738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:12.566476 23738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:12.601439 23738 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:44531
I20260927 04:59:12.601550 23865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:44531 every 8 connection(s)
I20260927 04:59:12.602739 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:12.609086 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 23738
I20260927 04:59:12.609277 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:12.609911 21157 external_mini_cluster.cc:1428] Setting key fbc33aab100d5c0f848662c139b93c38
Sep 27 04:59:12 dist-test-slave-39f2 krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485152, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:12.623764 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:12.605444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:55977 (local address 127.20.169.126:42941)
0927 04:59:12.605909 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:12.605914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:12.606740 (+ 826us) server_negotiation.cc:407] Connection header received
0927 04:59:12.608516 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:12.608521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:12.608634 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:12.608752 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:12.610478 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:12.611207 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:12.612966 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:12.613733 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:12.613895 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:12.617163 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:12.617168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:12.617172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:12.617202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:12.620267 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:12.621142 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:12.621150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:12.621157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:12.621300 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:12.621729 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:12.621737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:12.621743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:12.622125 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:12.622301 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:12.623043 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 04:59:12.623300 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":256,"threads_started":1}
I20260927 04:59:12.624420 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:12.605886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42941 (local address 127.20.169.65:55977)
0927 04:59:12.606579 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:12.606622 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:12.608249 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:12.609024 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:12.609032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:12.609661 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:12.610296 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:12.610316 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:12.611354 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:12.611359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:12.612768 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:12.612779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:12.613958 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:12.613963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:12.614152 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:12.614823 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:12.614845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:12.616935 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:12.620452 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:12.620460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:12.620472 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:12.621010 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:12.621421 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:12.621425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:12.621427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:12.621609 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:12.622304 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:12.622311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:12.622651 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 04:59:12.622876 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 04:59:12.623254 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":260,"threads_started":1}
I20260927 04:59:12.626209 23866 heartbeater.cc:347] Connected to a master server at 127.20.169.126:42941
I20260927 04:59:12.626600 23866 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:12.627390 23866 heartbeater.cc:511] Master 127.20.169.126:42941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 04:59:12.629675 23676 ts_manager.cc:194] Registered new tserver with Master: e1b5129042264ccaa273b451a2b00019 (127.20.169.65:44531)
I20260927 04:59:12.631466 23676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:55977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:12.657148 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42941
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:12.975018 23872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:12.975282 23872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:12.975384 23872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:12.983036 23872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:12.983202 23872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:12.983340 23872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:12.992767 23872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:12.994107 23872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:12.995370 23872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:13.011545 23881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:13.011587 23878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:13.012890 23872 server_base.cc:1034] running on GCE node
W20260927 04:59:13.013387 23877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:13.014215 23872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:13.015337 23872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:13.016571 23872 hybrid_clock.cc:648] HybridClock initialized: now 1790485153016512 us; error 66 us; skew 500 ppm
I20260927 04:59:13.016927 23872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485153, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:13.020037 23872 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:13.021490 23872 webserver.cc:460] Webserver started at http://127.20.169.66:43705/ using document root <none> and password file <none>
I20260927 04:59:13.022171 23872 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:13.022297 23872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:13.022614 23872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:13.025601 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "00385ecec57b46db81c38cc47e641a79"
format_stamp: "Formatted at 2026-09-27 04:59:13 on dist-test-slave-39f2"
server_key: "6ad1927c20fa82e2301dca97d7b0ef57"
server_key_iv: "0683c530899440a4e30cf89e1ebe92e2"
server_key_version: "encryptionkey@0"
I20260927 04:59:13.026383 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "00385ecec57b46db81c38cc47e641a79"
format_stamp: "Formatted at 2026-09-27 04:59:13 on dist-test-slave-39f2"
server_key: "6ad1927c20fa82e2301dca97d7b0ef57"
server_key_iv: "0683c530899440a4e30cf89e1ebe92e2"
server_key_version: "encryptionkey@0"
I20260927 04:59:13.031459 23872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 04:59:13.034830 23888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:13.036228 23872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:13.036463 23872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "00385ecec57b46db81c38cc47e641a79"
format_stamp: "Formatted at 2026-09-27 04:59:13 on dist-test-slave-39f2"
server_key: "6ad1927c20fa82e2301dca97d7b0ef57"
server_key_iv: "0683c530899440a4e30cf89e1ebe92e2"
server_key_version: "encryptionkey@0"
I20260927 04:59:13.036661 23872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:13.056223 23872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:13.060595 23872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:13.060976 23872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:13.061981 23872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:13.063429 23872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:13.063519 23872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:13.063728 23872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:13.063798 23872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:13.098325 23872 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:40491
I20260927 04:59:13.098377 24000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:40491 every 8 connection(s)
I20260927 04:59:13.099696 23872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:13.105532 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 23872
I20260927 04:59:13.105731 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:13.106388 21157 external_mini_cluster.cc:1428] Setting key 40fbb8560ad0a8c81a37e0bdfd9ac57d
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485153, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:13.122243 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:13.102554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:45777 (local address 127.20.169.126:42941)
0927 04:59:13.102780 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:13.102787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:13.104042 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 04:59:13.106028 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:13.106036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:13.106129 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:13.106268 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:13.107860 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.108666 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.110236 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.110654 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.110795 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:13.114349 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:13.114359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:13.114366 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:13.114419 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:13.118486 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:13.119300 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:13.119310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:13.119317 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:13.119495 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:13.119927 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:13.119935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:13.119941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:13.120374 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:13.120540 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:13.121361 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 04:59:13.121659 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 04:59:13.122910 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:13.103127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42941 (local address 127.20.169.66:45777)
0927 04:59:13.103871 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:13.103915 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:13.105686 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:13.106562 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:13.106579 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:13.107129 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:13.107677 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:13.107695 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.108848 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.108855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:13.110089 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:13.110100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.110897 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.110904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:13.111127 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:13.111802 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:13.111838 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:13.114051 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:13.118655 (+ 4604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:13.118668 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:13.118683 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:13.119125 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:13.119618 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:13.119621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:13.119623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:13.119797 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:13.120511 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:13.120517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:13.120930 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 04:59:13.121166 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 04:59:13.121613 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":281,"threads_started":1}
I20260927 04:59:13.124555 24001 heartbeater.cc:347] Connected to a master server at 127.20.169.126:42941
I20260927 04:59:13.125043 24001 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:13.125890 24001 heartbeater.cc:511] Master 127.20.169.126:42941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:59:13.127449 23676 ts_manager.cc:194] Registered new tserver with Master: 00385ecec57b46db81c38cc47e641a79 (127.20.169.66:40491)
I20260927 04:59:13.128298 23676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:45777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:13.154038 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42941
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:13.478595 24006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:13.478883 24006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:13.479059 24006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:13.487876 24006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:13.487982 24006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:13.488186 24006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:13.498747 24006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:13.500034 24006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:13.501422 24006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:13.515007 24011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:13.514986 24012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:13.515350 24014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:13.515771 24006 server_base.cc:1034] running on GCE node
I20260927 04:59:13.516448 24006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:13.517421 24006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:13.518621 24006 hybrid_clock.cc:648] HybridClock initialized: now 1790485153518560 us; error 59 us; skew 500 ppm
I20260927 04:59:13.518929 24006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485153, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:13.522017 24006 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:13.523519 24006 webserver.cc:460] Webserver started at http://127.20.169.67:33403/ using document root <none> and password file <none>
I20260927 04:59:13.524175 24006 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:13.524320 24006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:13.524626 24006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:13.527529 24006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "583e746797904d4d90bdf678105b4020"
format_stamp: "Formatted at 2026-09-27 04:59:13 on dist-test-slave-39f2"
server_key: "3a077d6b454d82bd26d9e3d076ffb239"
server_key_iv: "8dc6345dd0663c330351000a2a08a2b5"
server_key_version: "encryptionkey@0"
I20260927 04:59:13.528317 24006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "583e746797904d4d90bdf678105b4020"
format_stamp: "Formatted at 2026-09-27 04:59:13 on dist-test-slave-39f2"
server_key: "3a077d6b454d82bd26d9e3d076ffb239"
server_key_iv: "8dc6345dd0663c330351000a2a08a2b5"
server_key_version: "encryptionkey@0"
I20260927 04:59:13.533489 24006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 04:59:13.537171 24021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:13.538702 24006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:13.538866 24006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "583e746797904d4d90bdf678105b4020"
format_stamp: "Formatted at 2026-09-27 04:59:13 on dist-test-slave-39f2"
server_key: "3a077d6b454d82bd26d9e3d076ffb239"
server_key_iv: "8dc6345dd0663c330351000a2a08a2b5"
server_key_version: "encryptionkey@0"
I20260927 04:59:13.539043 24006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:13.555433 24006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:13.559513 24006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:13.559794 24006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:13.560757 24006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:13.562117 24006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:13.562188 24006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:13.562278 24006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:13.562321 24006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:13.597048 24006 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:45807
I20260927 04:59:13.597085 24134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:45807 every 8 connection(s)
I20260927 04:59:13.598397 24006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:13.602375 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24006
I20260927 04:59:13.602533 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:13.603084 21157 external_mini_cluster.cc:1428] Setting key 102d57416f67a8970cf3c9fa5cd59813
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485153, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:13.619840 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:13.601133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:34873 (local address 127.20.169.126:42941)
0927 04:59:13.601366 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:13.601372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:13.602687 (+ 1315us) server_negotiation.cc:407] Connection header received
0927 04:59:13.605255 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:13.605264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:13.605349 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:13.605458 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:13.607507 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.608259 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.609814 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.610243 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.610360 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:13.613720 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:13.613725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:13.613729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:13.613767 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:13.616629 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:13.617375 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:13.617382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:13.617385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:13.617546 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:13.618104 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:13.618108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:13.618110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:13.618378 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:13.618551 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:13.619292 (+ 741us) server_negotiation.cc:299] Negotiation successful
0927 04:59:13.619505 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 04:59:13.620852 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:13.601632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42941 (local address 127.20.169.67:34873)
0927 04:59:13.602530 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:13.602576 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:13.604891 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:13.605695 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:13.605706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:13.606440 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:13.607320 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:13.607347 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.608397 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.608401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:13.609664 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:13.609675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:13.610426 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:13.610431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:13.610637 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:13.611218 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:13.611239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:13.613486 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:13.616765 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:13.616771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:13.616783 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:13.617238 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:13.617693 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:13.617698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:13.617702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:13.617970 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:13.618525 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:13.618532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:13.618923 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 04:59:13.619182 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 04:59:13.619690 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":319,"threads_started":1}
I20260927 04:59:13.622253 24135 heartbeater.cc:347] Connected to a master server at 127.20.169.126:42941
I20260927 04:59:13.622579 24135 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:13.623296 24135 heartbeater.cc:511] Master 127.20.169.126:42941 requested a full tablet report, sending...
I20260927 04:59:13.624703 23676 ts_manager.cc:194] Registered new tserver with Master: 583e746797904d4d90bdf678105b4020 (127.20.169.67:45807)
I20260927 04:59:13.625561 23676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:34873
I20260927 04:59:13.630921 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:59:13.634593 23866 heartbeater.cc:503] Master 127.20.169.126:42941 was elected leader, sending a full tablet report...
W20260927 04:59:13.662536 23868 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:13.659993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.20.169.126:42941)
0927 04:59:13.660185 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:13.660190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:13.660683 (+ 493us) server_negotiation.cc:407] Connection header received
0927 04:59:13.660802 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:13.660857 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:13.660952 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:13.661051 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:13.662332 (+ 1281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46720: BlockingRecv error: recv got EOF from 127.0.0.1:46720 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 04:59:13.665980 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23738
W20260927 04:59:13.668928 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23738: Not authorized: Client connection negotiation failed: client connection to 127.20.169.65:44531: server requires authentication, but client does not have Kerberos credentials available
I20260927 04:59:13.669137 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 23738
I20260927 04:59:13.685487 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23872
W20260927 04:59:13.688182 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23872: Not authorized: Client connection negotiation failed: client connection to 127.20.169.66:40491: server requires authentication, but client does not have Kerberos credentials available
I20260927 04:59:13.688333 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 23872
I20260927 04:59:13.704314 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24006
W20260927 04:59:13.707036 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24006: Not authorized: Client connection negotiation failed: client connection to 127.20.169.67:45807: server requires authentication, but client does not have Kerberos credentials available
I20260927 04:59:13.707201 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24006
I20260927 04:59:13.724447 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 23644
I20260927 04:59:13.725631 23706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:13.827201 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 23644
2026-09-27T04:59:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3568 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[24152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[24152](info): set up 2 sockets
Sep 27 04:59:13 dist-test-slave-39f2 krb5kdc[24152](info): commencing operation
krb5kdc: starting...
W20260927 04:59:14.932749 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:15 dist-test-slave-39f2 krb5kdc[24152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:15.128916 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45445
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:15.445218 24165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:15.445533 24165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:15.445721 24165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:15.453591 24165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:15.453691 24165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:15.453802 24165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:15.453899 24165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:15.453965 24165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:15.463696 24165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:15.464986 24165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:15.466245 24165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:15.478026 24173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:15.478116 24170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:15.478055 24171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:15.479074 24165 server_base.cc:1034] running on GCE node
I20260927 04:59:15.479923 24165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:15.481438 24165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:15.482656 24165 hybrid_clock.cc:648] HybridClock initialized: now 1790485155482608 us; error 33 us; skew 500 ppm
I20260927 04:59:15.483052 24165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:15 dist-test-slave-39f2 krb5kdc[24152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485155, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:15.492074 24165 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:15.493853 24165 webserver.cc:460] Webserver started at http://127.20.169.126:44733/ using document root <none> and password file <none>
I20260927 04:59:15.494629 24165 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:15.494789 24165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:15.495113 24165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:15.498637 24165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c"
format_stamp: "Formatted at 2026-09-27 04:59:15 on dist-test-slave-39f2"
server_key: "48e2e726680641e191540030bdd6312b"
server_key_iv: "0a80a87d0c66e25a6dc186604c939e16"
server_key_version: "encryptionkey@0"
I20260927 04:59:15.499469 24165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c"
format_stamp: "Formatted at 2026-09-27 04:59:15 on dist-test-slave-39f2"
server_key: "48e2e726680641e191540030bdd6312b"
server_key_iv: "0a80a87d0c66e25a6dc186604c939e16"
server_key_version: "encryptionkey@0"
I20260927 04:59:15.505348 24165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 04:59:15.509151 24180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:15.510592 24165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:15.510819 24165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c"
format_stamp: "Formatted at 2026-09-27 04:59:15 on dist-test-slave-39f2"
server_key: "48e2e726680641e191540030bdd6312b"
server_key_iv: "0a80a87d0c66e25a6dc186604c939e16"
server_key_version: "encryptionkey@0"
I20260927 04:59:15.511014 24165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:15.531986 24165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:15.536988 24165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:15.537333 24165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:15.558691 24165 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45445
I20260927 04:59:15.558738 24231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45445 every 8 connection(s)
I20260927 04:59:15.560261 24165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:15.564136 24232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:15.566203 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24165
I20260927 04:59:15.566519 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:15.567215 21157 external_mini_cluster.cc:1428] Setting key 62c8cd0c422c6bcbbb7e2a1a97fc1b01
I20260927 04:59:15.576908 24232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c: Bootstrap starting.
I20260927 04:59:15.580391 24232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:15.581398 24232 log.cc:824] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c: Log is configured to *not* fsync() on all Append() calls
Sep 27 04:59:15 dist-test-slave-39f2 krb5kdc[24152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:15.584232 24232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c: No bootstrap required, opened a new log
I20260927 04:59:15.593178 24232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45445 } }
I20260927 04:59:15.593562 24232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:15.593614 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:15.569529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.20.169.126:45445)
0927 04:59:15.570379 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:15.570394 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:15.570441 (+ 47us) server_negotiation.cc:407] Connection header received
0927 04:59:15.572793 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:15.572847 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:15.573343 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:15.573787 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:15.575392 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:15.577354 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:15.578779 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:15.579854 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:15.580021 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:15.584739 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:15.584748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:15.584766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:15.584832 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:15.589037 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:15.589814 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:15.589854 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:15.589860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:15.590045 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:15.590735 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:15.590743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:15.590746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:15.591353 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:15.591678 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:15.592125 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 04:59:15.592549 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":378,"threads_started":1}
I20260927 04:59:15.593680 24232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ba032fa19c4d7b8d1aa8c0da0f523c, State: Initialized, Role: FOLLOWER
I20260927 04:59:15.594348 24232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45445 } }
I20260927 04:59:15.594592 24232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:15.594775 24232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:15.594967 24232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:15.597223 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:59:15.597538 24232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45445 } }
I20260927 04:59:15.598008 24232 leader_election.cc:302] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55ba032fa19c4d7b8d1aa8c0da0f523c; no voters:
I20260927 04:59:15.598551 24232 leader_election.cc:288] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:15.598688 24237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:15.599852 24237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [term 1 LEADER]: Becoming Leader. State: Replica: 55ba032fa19c4d7b8d1aa8c0da0f523c, State: Running, Role: LEADER
I20260927 04:59:15.600539 24232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:15.600454 24237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45445 } }
I20260927 04:59:15.607095 24239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55ba032fa19c4d7b8d1aa8c0da0f523c. Latest consensus state: current_term: 1 leader_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45445 } } }
I20260927 04:59:15.606822 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ba032fa19c4d7b8d1aa8c0da0f523c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45445 } } }
I20260927 04:59:15.607468 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:15.607468 24239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:15.623880 24248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:15.628768 24248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:15.637154 24248 catalog_manager.cc:1347] Generated new cluster ID: 1821d0b7cc79480285f4587e6e7d520f
I20260927 04:59:15.637307 24248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:15.654521 24248 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:15.655957 24248 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:15.656014 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:15.646823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.20.169.126:45445)
0927 04:59:15.646988 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:15.646994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:15.647601 (+ 607us) server_negotiation.cc:407] Connection header received
0927 04:59:15.647773 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:15.647779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:15.647863 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:15.647999 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:15.649130 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:15.650256 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:15.652070 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:15.652709 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:15.652856 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:15.654083 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:15.654090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:15.654095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:15.655003 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:15.655088 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:15.655123 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:15.655244 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:15.655387 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 04:59:15.655484 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 04:59:15.664278 24248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55ba032fa19c4d7b8d1aa8c0da0f523c: Generated new TSK 0
I20260927 04:59:15.665301 24248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:59:15.677346 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24165
I20260927 04:59:15.679337 24227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:15.768435 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24165
2026-09-27T04:59:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1935 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:15 dist-test-slave-39f2 krb5kdc[24266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:15 dist-test-slave-39f2 krb5kdc[24266](info): set up 2 sockets
Sep 27 04:59:15 dist-test-slave-39f2 krb5kdc[24266](info): commencing operation
krb5kdc: starting...
W20260927 04:59:16.864722 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:17 dist-test-slave-39f2 krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:17.084555 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45203
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:17.412822 24280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:17.413164 24280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:17.413353 24280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:17.420385 24280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:17.420535 24280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:17.420643 24280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:17.420770 24280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:17.420888 24280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:17.430563 24280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:17.431893 24280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:17.433264 24280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:17.446110 24285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:17.446127 24286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:17.446127 24288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:17.446893 24280 server_base.cc:1034] running on GCE node
I20260927 04:59:17.447646 24280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:17.449100 24280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:17.450305 24280 hybrid_clock.cc:648] HybridClock initialized: now 1790485157450259 us; error 64 us; skew 500 ppm
I20260927 04:59:17.450644 24280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:17 dist-test-slave-39f2 krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485157, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:17.453954 24280 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:17.455613 24280 webserver.cc:460] Webserver started at http://127.20.169.126:43357/ using document root <none> and password file <none>
I20260927 04:59:17.456326 24280 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:17.456485 24280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:17.456869 24280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:17.459968 24280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "74f39b5d79534e56b354b34f7ac8c65f"
format_stamp: "Formatted at 2026-09-27 04:59:17 on dist-test-slave-39f2"
server_key: "f51fba65e56e3d033f1fdcf2757e2989"
server_key_iv: "564327b1a9bf1e227382b3fcf69dd1b9"
server_key_version: "encryptionkey@0"
I20260927 04:59:17.460791 24280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "74f39b5d79534e56b354b34f7ac8c65f"
format_stamp: "Formatted at 2026-09-27 04:59:17 on dist-test-slave-39f2"
server_key: "f51fba65e56e3d033f1fdcf2757e2989"
server_key_iv: "564327b1a9bf1e227382b3fcf69dd1b9"
server_key_version: "encryptionkey@0"
I20260927 04:59:17.466216 24280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 04:59:17.469985 24295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:17.471458 24280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 04:59:17.471654 24280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "74f39b5d79534e56b354b34f7ac8c65f"
format_stamp: "Formatted at 2026-09-27 04:59:17 on dist-test-slave-39f2"
server_key: "f51fba65e56e3d033f1fdcf2757e2989"
server_key_iv: "564327b1a9bf1e227382b3fcf69dd1b9"
server_key_version: "encryptionkey@0"
I20260927 04:59:17.471873 24280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:17.498663 24280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:17.503408 24280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:17.503698 24280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:17.522814 24280 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45203
I20260927 04:59:17.522841 24346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45203 every 8 connection(s)
I20260927 04:59:17.524125 24280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:17.527846 24347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:17.533649 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24280
I20260927 04:59:17.533900 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:17.534693 21157 external_mini_cluster.cc:1428] Setting key df35904fcf4417291535f6d85f5403a3
I20260927 04:59:17.539036 24347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f: Bootstrap starting.
I20260927 04:59:17.542032 24347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:17.543020 24347 log.cc:824] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:17.545996 24347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f: No bootstrap required, opened a new log
Sep 27 04:59:17 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:17.555195 24347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f39b5d79534e56b354b34f7ac8c65f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45203 } }
I20260927 04:59:17.555645 24347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:17.555753 24347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f39b5d79534e56b354b34f7ac8c65f, State: Initialized, Role: FOLLOWER
I20260927 04:59:17.556495 24347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f39b5d79534e56b354b34f7ac8c65f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45203 } }
I20260927 04:59:17.556778 24347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:17.556981 24347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:17.557173 24347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:17.558780 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:17.536957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.20.169.126:45203)
0927 04:59:17.537697 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:17.537709 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:17.537748 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:59:17.539532 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:17.539554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:17.539925 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:17.540326 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:17.541749 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:17.543141 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:17.544800 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:17.545521 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:17.545732 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:17.550159 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:17.550167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:17.550182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:17.550226 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:17.553778 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:17.554668 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:17.554675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:17.554682 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:17.554872 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:17.555394 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:17.555398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:17.555400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:17.556182 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:17.556451 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:17.557094 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 04:59:17.557646 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":335,"threads_started":1}
I20260927 04:59:17.559655 24347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f39b5d79534e56b354b34f7ac8c65f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45203 } }
I20260927 04:59:17.560155 24347 leader_election.cc:302] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f39b5d79534e56b354b34f7ac8c65f; no voters:
I20260927 04:59:17.560751 24347 leader_election.cc:288] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:17.560981 24352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:17.562384 24352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [term 1 LEADER]: Becoming Leader. State: Replica: 74f39b5d79534e56b354b34f7ac8c65f, State: Running, Role: LEADER
I20260927 04:59:17.562981 24352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f39b5d79534e56b354b34f7ac8c65f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45203 } }
I20260927 04:59:17.563247 24347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:17.566134 24353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f39b5d79534e56b354b34f7ac8c65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f39b5d79534e56b354b34f7ac8c65f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45203 } } }
I20260927 04:59:17.566169 24354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f39b5d79534e56b354b34f7ac8c65f. Latest consensus state: current_term: 1 leader_uuid: "74f39b5d79534e56b354b34f7ac8c65f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f39b5d79534e56b354b34f7ac8c65f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45203 } } }
I20260927 04:59:17.566634 24354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:17.566524 24353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:17.567588 24358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:17.572901 24358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:17.583304 24358 catalog_manager.cc:1347] Generated new cluster ID: 0d4b0efa941441258fe7e0f5baa013df
I20260927 04:59:17.583439 24358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:17.599539 24358 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:17.600898 24358 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:17.612049 24358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f39b5d79534e56b354b34f7ac8c65f: Generated new TSK 0
I20260927 04:59:17.612901 24358 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:17.670603 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45203
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:18.014773 24373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:18.015072 24373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:18.015167 24373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:18.022476 24373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:18.022590 24373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:18.022703 24373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:18.032467 24373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:18.034029 24373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:18.035372 24373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:18.048255 24379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:18.048403 24378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:18.048313 24381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:18.049145 24373 server_base.cc:1034] running on GCE node
I20260927 04:59:18.049834 24373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:18.050837 24373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:18.052112 24373 hybrid_clock.cc:648] HybridClock initialized: now 1790485158052065 us; error 56 us; skew 500 ppm
I20260927 04:59:18.052487 24373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:18 dist-test-slave-39f2 krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:18.056118 24373 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:18.057842 24373 webserver.cc:460] Webserver started at http://127.20.169.65:33897/ using document root <none> and password file <none>
I20260927 04:59:18.058573 24373 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:18.058737 24373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:18.059074 24373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:18.062326 24373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "6aeac11fc9dc4e09a000defc2f478746"
format_stamp: "Formatted at 2026-09-27 04:59:18 on dist-test-slave-39f2"
server_key: "7c0e4f6d56650d5a77527d165cf74dbb"
server_key_iv: "3f6be1724db661a730a7baad4f66ac8d"
server_key_version: "encryptionkey@0"
I20260927 04:59:18.063246 24373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "6aeac11fc9dc4e09a000defc2f478746"
format_stamp: "Formatted at 2026-09-27 04:59:18 on dist-test-slave-39f2"
server_key: "7c0e4f6d56650d5a77527d165cf74dbb"
server_key_iv: "3f6be1724db661a730a7baad4f66ac8d"
server_key_version: "encryptionkey@0"
I20260927 04:59:18.069057 24373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 04:59:18.072751 24389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:18.074229 24373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:18.074427 24373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "6aeac11fc9dc4e09a000defc2f478746"
format_stamp: "Formatted at 2026-09-27 04:59:18 on dist-test-slave-39f2"
server_key: "7c0e4f6d56650d5a77527d165cf74dbb"
server_key_iv: "3f6be1724db661a730a7baad4f66ac8d"
server_key_version: "encryptionkey@0"
I20260927 04:59:18.074643 24373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:18.087417 24373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:18.092365 24373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:18.092700 24373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:18.093745 24373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:18.095203 24373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:18.095323 24373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:18.095482 24373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:18.095566 24373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:18.129925 24373 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:42511
I20260927 04:59:18.129946 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:42511 every 8 connection(s)
I20260927 04:59:18.131417 24373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:18.133144 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24373
I20260927 04:59:18.133386 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:18.134101 21157 external_mini_cluster.cc:1428] Setting key 562465477c4f27705d78573c76dd6791
Sep 27 04:59:18 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:18.155601 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:18.134330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:47797 (local address 127.20.169.126:45203)
0927 04:59:18.134853 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:18.134861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:18.135913 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 04:59:18.138624 (+ 2711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:18.138628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:18.138692 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:18.138805 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:18.140923 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.141668 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.143713 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.144504 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.144654 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:18.148048 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:18.148057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:18.148065 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:18.148112 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:18.152057 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:18.152856 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:18.152865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:18.152870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:18.153043 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:18.153501 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:18.153507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:18.153511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:18.153931 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:18.154100 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:18.154833 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 04:59:18.155087 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 04:59:18.156203 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:18.134913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45203 (local address 127.20.169.65:47797)
0927 04:59:18.135752 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:18.135805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:18.138290 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:18.139060 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:18.139073 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:18.139866 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:18.140685 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:18.140715 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.141826 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.141834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:18.143551 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:18.143568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.144726 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.144731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:18.144997 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:18.145717 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:18.145737 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:18.147817 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:18.152202 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:18.152209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:18.152223 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:18.152670 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:18.153169 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:18.153172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:18.153175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:18.153362 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:18.154071 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:18.154077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:18.154408 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 04:59:18.154632 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 04:59:18.155039 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":345,"threads_started":1}
I20260927 04:59:18.157960 24502 heartbeater.cc:347] Connected to a master server at 127.20.169.126:45203
I20260927 04:59:18.158295 24502 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 04:59:18.159265 24502 heartbeater.cc:511] Master 127.20.169.126:45203 requested a full tablet report, sending...
I20260927 04:59:18.161650 24312 ts_manager.cc:194] Registered new tserver with Master: 6aeac11fc9dc4e09a000defc2f478746 (127.20.169.65:42511)
I20260927 04:59:18.163215 24312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:47797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:18.184039 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45203
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:18.507392 24508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:18.507717 24508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:18.507921 24508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:18.515172 24508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:18.515273 24508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:18.515383 24508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:18.526695 24508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:18.528123 24508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:18.529764 24508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:18.545167 24513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:18.545109 24514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:18.545059 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:18.546094 24508 server_base.cc:1034] running on GCE node
I20260927 04:59:18.546743 24508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:18.547747 24508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:18.548983 24508 hybrid_clock.cc:648] HybridClock initialized: now 1790485158548910 us; error 63 us; skew 500 ppm
I20260927 04:59:18.549304 24508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:18 dist-test-slave-39f2 krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:18.552740 24508 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:18.554207 24508 webserver.cc:460] Webserver started at http://127.20.169.66:35363/ using document root <none> and password file <none>
I20260927 04:59:18.554899 24508 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:18.555047 24508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:18.555346 24508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:18.558277 24508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "33793f5d158548ca874a53f354d65d9f"
format_stamp: "Formatted at 2026-09-27 04:59:18 on dist-test-slave-39f2"
server_key: "6ee86eb35ed2793d9de5d61c91deeb2f"
server_key_iv: "60915a58dc148c14921a097e89027b32"
server_key_version: "encryptionkey@0"
I20260927 04:59:18.559054 24508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "33793f5d158548ca874a53f354d65d9f"
format_stamp: "Formatted at 2026-09-27 04:59:18 on dist-test-slave-39f2"
server_key: "6ee86eb35ed2793d9de5d61c91deeb2f"
server_key_iv: "60915a58dc148c14921a097e89027b32"
server_key_version: "encryptionkey@0"
I20260927 04:59:18.564393 24508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 04:59:18.567770 24524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:18.569161 24508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:18.569339 24508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "33793f5d158548ca874a53f354d65d9f"
format_stamp: "Formatted at 2026-09-27 04:59:18 on dist-test-slave-39f2"
server_key: "6ee86eb35ed2793d9de5d61c91deeb2f"
server_key_iv: "60915a58dc148c14921a097e89027b32"
server_key_version: "encryptionkey@0"
I20260927 04:59:18.569525 24508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:18.588755 24508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:18.592852 24508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:18.593149 24508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:18.594087 24508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:18.595427 24508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:18.595534 24508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:18.595707 24508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:18.595793 24508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:18.630147 24508 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:43209
I20260927 04:59:18.630189 24637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:43209 every 8 connection(s)
I20260927 04:59:18.631477 24508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:18.632143 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24508
I20260927 04:59:18.632448 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:18.633062 21157 external_mini_cluster.cc:1428] Setting key 44c2449974f85317b7cffc36bbf4c105
Sep 27 04:59:18 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485158, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:18.654134 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:18.634489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40045 (local address 127.20.169.126:45203)
0927 04:59:18.634683 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:18.634689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:18.636079 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 04:59:18.637940 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:18.637945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:18.638033 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:18.638142 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:18.639650 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.640369 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.642126 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.642542 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.642678 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:18.646367 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:18.646376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:18.646385 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:18.646435 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:18.650466 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:18.651526 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:18.651530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:18.651533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:18.651675 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:18.652200 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:18.652204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:18.652206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:18.652576 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:18.652699 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:18.653677 (+ 978us) server_negotiation.cc:299] Negotiation successful
0927 04:59:18.653878 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:59:18.655481 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:18.635110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45203 (local address 127.20.169.66:40045)
0927 04:59:18.635933 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:18.635971 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:18.637679 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:18.638396 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:18.638404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:18.638911 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:18.639486 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:18.639506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.640514 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.640525 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:18.641989 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:18.642002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:18.642767 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:18.642774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:18.643018 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:18.643700 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:18.643728 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:18.646120 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:18.650636 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:18.650648 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:18.650667 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:18.651367 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:18.651797 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:18.651804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:18.651810 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:18.652065 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:18.652714 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:18.652722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:18.653238 (+ 516us) client_negotiation.cc:769] Sending connection context
0927 04:59:18.653571 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 04:59:18.654128 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 04:59:18.657238 24638 heartbeater.cc:347] Connected to a master server at 127.20.169.126:45203
I20260927 04:59:18.657564 24638 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:18.658380 24638 heartbeater.cc:511] Master 127.20.169.126:45203 requested a full tablet report, sending...
I20260927 04:59:18.660045 24312 ts_manager.cc:194] Registered new tserver with Master: 33793f5d158548ca874a53f354d65d9f (127.20.169.66:43209)
I20260927 04:59:18.661392 24312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:40045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:18.684633 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45203
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:19.001647 24643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:19.001906 24643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:19.002069 24643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:19.009445 24643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:19.009555 24643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:19.009734 24643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:19.019232 24643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:19.020516 24643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:19.021831 24643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:19.034371 24648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:19.034526 24649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:19.034868 24652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:19.035763 24643 server_base.cc:1034] running on GCE node
I20260927 04:59:19.036389 24643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:19.037345 24643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:19.038537 24643 hybrid_clock.cc:648] HybridClock initialized: now 1790485159038493 us; error 35 us; skew 500 ppm
I20260927 04:59:19.038852 24643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:19.041926 24643 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:19.043499 24643 webserver.cc:460] Webserver started at http://127.20.169.67:44495/ using document root <none> and password file <none>
I20260927 04:59:19.044109 24643 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:19.044231 24643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:19.044587 24643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:19.047407 24643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "53d010e901b84fbeac681f438a73d339"
format_stamp: "Formatted at 2026-09-27 04:59:19 on dist-test-slave-39f2"
server_key: "3d5e4600696c757295b4c9a3ea06e3f5"
server_key_iv: "f9969849c18ab01d3bad8d5359663bfd"
server_key_version: "encryptionkey@0"
I20260927 04:59:19.048102 24643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "53d010e901b84fbeac681f438a73d339"
format_stamp: "Formatted at 2026-09-27 04:59:19 on dist-test-slave-39f2"
server_key: "3d5e4600696c757295b4c9a3ea06e3f5"
server_key_iv: "f9969849c18ab01d3bad8d5359663bfd"
server_key_version: "encryptionkey@0"
I20260927 04:59:19.053164 24643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 04:59:19.056464 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:19.057938 24643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:19.058101 24643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "53d010e901b84fbeac681f438a73d339"
format_stamp: "Formatted at 2026-09-27 04:59:19 on dist-test-slave-39f2"
server_key: "3d5e4600696c757295b4c9a3ea06e3f5"
server_key_iv: "f9969849c18ab01d3bad8d5359663bfd"
server_key_version: "encryptionkey@0"
I20260927 04:59:19.058290 24643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:19.094184 24643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:19.098253 24643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:19.098569 24643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:19.099571 24643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:19.100944 24643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:19.101049 24643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:19.101202 24643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:19.101274 24643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:19.135495 24643 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:34521
I20260927 04:59:19.135530 24772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:34521 every 8 connection(s)
I20260927 04:59:19.136855 24643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:19.144069 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24643
I20260927 04:59:19.144289 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:19.144706 21157 external_mini_cluster.cc:1428] Setting key 17746c2a43465f58bf9ee389c02cc9df
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:19.156773 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:19.139415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:54069 (local address 127.20.169.126:45203)
0927 04:59:19.139676 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:19.139683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:19.140690 (+ 1007us) server_negotiation.cc:407] Connection header received
0927 04:59:19.142549 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:19.142554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:19.142641 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:19.142753 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:19.144189 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.145071 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.146962 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.147455 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.147585 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:19.151166 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:19.151172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:19.151175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:19.151208 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:19.153840 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.154533 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.154541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.154551 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.154680 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.155128 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.155134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.155137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.155393 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:19.155545 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:19.156207 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 04:59:19.156423 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 04:59:19.157711 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:19.139871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45203 (local address 127.20.169.67:54069)
0927 04:59:19.140539 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:19.140574 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:19.142255 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:19.142996 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:19.143004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:19.143479 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:19.144010 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:19.144026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.145221 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.145234 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:19.146829 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:19.146840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.147674 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.147685 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:19.147921 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:19.148571 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:19.148599 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:19.150948 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:19.153981 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:19.153990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:19.154012 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:19.154410 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:19.154788 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:19.154793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:19.154797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:19.155018 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:19.155516 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:19.155523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:19.155844 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 04:59:19.156125 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 04:59:19.156632 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":259,"threads_started":1}
I20260927 04:59:19.159375 24773 heartbeater.cc:347] Connected to a master server at 127.20.169.126:45203
I20260927 04:59:19.159896 24773 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:19.160984 24773 heartbeater.cc:511] Master 127.20.169.126:45203 requested a full tablet report, sending...
I20260927 04:59:19.162822 24311 ts_manager.cc:194] Registered new tserver with Master: 53d010e901b84fbeac681f438a73d339 (127.20.169.67:34521)
I20260927 04:59:19.163892 24311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:54069
I20260927 04:59:19.166476 24502 heartbeater.cc:503] Master 127.20.169.126:45203 was elected leader, sending a full tablet report...
I20260927 04:59:19.172631 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:19.223549 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:19.209122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.20.169.126:45203)
0927 04:59:19.209307 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:19.209312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:19.209796 (+ 484us) server_negotiation.cc:407] Connection header received
0927 04:59:19.209961 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:19.209965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:19.210038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:19.210128 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:19.211463 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.212213 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.213529 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.213945 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.214042 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:19.217889 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:19.217895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:19.217899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:19.217940 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:19.221138 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.221744 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.221748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.221751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.221904 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.222257 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.222260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.222263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.222667 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:19.222890 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:19.223094 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 04:59:19.223294 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 04:59:19.224225 24311 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44766
I20260927 04:59:19.226120 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24373
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:19.242004 24784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:19.227107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.20.169.65:42511)
0927 04:59:19.227713 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:19.227720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:19.227744 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:59:19.227797 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:19.227803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:19.228013 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:19.228152 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:19.229576 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.230635 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.231803 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.232381 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.232475 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:19.236512 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:19.236523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:19.236536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:19.236571 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:19.239399 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.239998 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.240005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.240007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.240120 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.240494 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.240499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.240501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.240927 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:19.241152 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:19.241514 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 04:59:19.241710 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":304,"threads_started":1}
W20260927 04:59:19.243382 24497 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42180
W20260927 04:59:19.244728 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24373: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:19.244946 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24373
I20260927 04:59:19.261840 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24508
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:19.278496 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:19.262769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.20.169.66:43209)
0927 04:59:19.263362 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:19.263369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:19.263393 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:59:19.263453 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:19.263459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:19.263737 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:19.263865 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:19.265241 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.266368 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.267660 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.268211 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.268351 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:19.272477 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:19.272487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:19.272506 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:19.272539 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:19.275683 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.276406 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.276413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.276415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.276541 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.276934 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.276941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.276947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.277313 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:19.277564 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:19.277879 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 04:59:19.278101 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":304,"threads_started":1}
W20260927 04:59:19.279814 24633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44980
W20260927 04:59:19.280997 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24508: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:19.281136 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24508
I20260927 04:59:19.297328 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24643
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485159, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:19.313781 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:19.298372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.20.169.67:34521)
0927 04:59:19.299024 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:19.299031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:19.299059 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:59:19.299117 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:19.299123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:19.299319 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:19.299499 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:19.300904 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.301910 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.303156 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:19.303746 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:19.303889 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:19.308214 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:19.308224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:19.308239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:19.308271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:19.310990 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.311643 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.311650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.311652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.311770 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:19.312133 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:19.312140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:19.312143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:19.312479 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:19.312808 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:19.313219 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 04:59:19.313482 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":316,"threads_started":1}
W20260927 04:59:19.315142 24768 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51510
W20260927 04:59:19.316229 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24643: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 04:59:19.316421 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24643
I20260927 04:59:19.332468 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24280
I20260927 04:59:19.333707 24342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:19.435899 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24280
2026-09-27T04:59:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3672 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24790](info): set up 2 sockets
Sep 27 04:59:19 dist-test-slave-39f2 krb5kdc[24790](info): commencing operation
krb5kdc: starting...
W20260927 04:59:20.545495 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:20 dist-test-slave-39f2 krb5kdc[24790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:20.746279 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33281
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:33281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:21.075515 24803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:21.075825 24803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:21.076010 24803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:21.083441 24803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:21.083549 24803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:21.083636 24803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:21.083730 24803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:21.083829 24803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:21.094874 24803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:33281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:21.096352 24803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:21.097798 24803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:21.109827 24811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:21.109875 24809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:21.110289 24808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:21.110957 24803 server_base.cc:1034] running on GCE node
I20260927 04:59:21.111796 24803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:21.113330 24803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:21.114593 24803 hybrid_clock.cc:648] HybridClock initialized: now 1790485161114528 us; error 50 us; skew 500 ppm
I20260927 04:59:21.114979 24803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:21 dist-test-slave-39f2 krb5kdc[24790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485161, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:21.124150 24803 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:21.125713 24803 webserver.cc:460] Webserver started at http://127.20.169.126:41221/ using document root <none> and password file <none>
I20260927 04:59:21.126471 24803 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:21.126757 24803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:21.127137 24803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:21.130398 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "ccde9393c9e646f6a620c12a44a6e833"
format_stamp: "Formatted at 2026-09-27 04:59:21 on dist-test-slave-39f2"
server_key: "47ad2d4ba4e044923d5c06d9f302bbd4"
server_key_iv: "e2d24d5d45a206fa5ab5bf78f8b103a9"
server_key_version: "encryptionkey@0"
I20260927 04:59:21.131314 24803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "ccde9393c9e646f6a620c12a44a6e833"
format_stamp: "Formatted at 2026-09-27 04:59:21 on dist-test-slave-39f2"
server_key: "47ad2d4ba4e044923d5c06d9f302bbd4"
server_key_iv: "e2d24d5d45a206fa5ab5bf78f8b103a9"
server_key_version: "encryptionkey@0"
I20260927 04:59:21.136893 24803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 04:59:21.140388 24818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:21.141695 24803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 04:59:21.141903 24803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "ccde9393c9e646f6a620c12a44a6e833"
format_stamp: "Formatted at 2026-09-27 04:59:21 on dist-test-slave-39f2"
server_key: "47ad2d4ba4e044923d5c06d9f302bbd4"
server_key_iv: "e2d24d5d45a206fa5ab5bf78f8b103a9"
server_key_version: "encryptionkey@0"
I20260927 04:59:21.142097 24803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:21.162648 24803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:21.167181 24803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:21.167478 24803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:21.186259 24803 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:33281
I20260927 04:59:21.186292 24870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:33281 every 8 connection(s)
I20260927 04:59:21.187584 24803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:21.191386 24871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:21.195324 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24803
I20260927 04:59:21.195538 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:21.196161 21157 external_mini_cluster.cc:1428] Setting key 6d8707618eca6eb817762cf3d92891fe
I20260927 04:59:21.203586 24871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833: Bootstrap starting.
I20260927 04:59:21.207629 24871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:21.208608 24871 log.cc:824] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:21.212096 24871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833: No bootstrap required, opened a new log
I20260927 04:59:21.221187 24871 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde9393c9e646f6a620c12a44a6e833" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33281 } }
I20260927 04:59:21.221599 24871 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:21.221689 24871 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccde9393c9e646f6a620c12a44a6e833, State: Initialized, Role: FOLLOWER
I20260927 04:59:21.222368 24871 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde9393c9e646f6a620c12a44a6e833" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33281 } }
I20260927 04:59:21.222644 24871 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:21.222777 24871 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:21.222956 24871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 04:59:21 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:21.226536 24871 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde9393c9e646f6a620c12a44a6e833" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33281 } }
I20260927 04:59:21.227185 24871 leader_election.cc:302] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccde9393c9e646f6a620c12a44a6e833; no voters:
I20260927 04:59:21.227943 24871 leader_election.cc:288] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:21.228073 24876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:21.235862 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:21.198089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.20.169.126:33281)
0927 04:59:21.198903 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:21.198917 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:21.198970 (+ 53us) server_negotiation.cc:407] Connection header received
0927 04:59:21.201751 (+ 2781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:21.201780 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:21.202249 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:21.202695 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:21.204836 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:21.207003 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:21.209048 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:21.209883 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:21.210153 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:21.225123 (+ 14970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:21.225134 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:21.225154 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:21.225204 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:21.229150 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:21.230231 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:21.230241 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:21.230250 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:21.230482 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:21.230984 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:21.230990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:21.230995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:21.231841 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:21.233454 (+ 1613us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:21.233766 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 04:59:21.234259 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":316,"threads_started":1}
I20260927 04:59:21.237092 24871 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:21.238875 24876 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [term 1 LEADER]: Becoming Leader. State: Replica: ccde9393c9e646f6a620c12a44a6e833, State: Running, Role: LEADER
I20260927 04:59:21.239457 24876 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde9393c9e646f6a620c12a44a6e833" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33281 } }
I20260927 04:59:21.243835 24877 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccde9393c9e646f6a620c12a44a6e833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde9393c9e646f6a620c12a44a6e833" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33281 } } }
I20260927 04:59:21.244232 24877 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:21.245810 24878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccde9393c9e646f6a620c12a44a6e833. Latest consensus state: current_term: 1 leader_uuid: "ccde9393c9e646f6a620c12a44a6e833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde9393c9e646f6a620c12a44a6e833" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33281 } } }
I20260927 04:59:21.246161 24878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:21.248754 24885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:21.254251 24885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:21.264120 24885 catalog_manager.cc:1347] Generated new cluster ID: 01220269d43349e4a0c3db049a60fdfd
I20260927 04:59:21.264226 24885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:21.279458 24885 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:21.281466 24885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:21.292645 24885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccde9393c9e646f6a620c12a44a6e833: Generated new TSK 0
I20260927 04:59:21.293602 24885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:21.352679 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33281
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:21.685741 24898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:21.686015 24898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:21.686175 24898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:21.693372 24898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:21.693464 24898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:21.693660 24898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:21.703156 24898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:21.704443 24898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:21.705780 24898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:21.718441 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:21.718492 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:21.718487 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:21.719391 24898 server_base.cc:1034] running on GCE node
I20260927 04:59:21.719941 24898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:21.720855 24898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:21.722085 24898 hybrid_clock.cc:648] HybridClock initialized: now 1790485161722033 us; error 61 us; skew 500 ppm
I20260927 04:59:21.722383 24898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:21 dist-test-slave-39f2 krb5kdc[24790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485161, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:21.725561 24898 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:21.726946 24898 webserver.cc:460] Webserver started at http://127.20.169.65:44737/ using document root <none> and password file <none>
I20260927 04:59:21.727579 24898 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:21.727717 24898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:21.728026 24898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:21.730820 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "d8521f8295d3486995ada4641bc581ca"
format_stamp: "Formatted at 2026-09-27 04:59:21 on dist-test-slave-39f2"
server_key: "fc662825f1bd5f9f79bc06fc4aa083b4"
server_key_iv: "6044b3e2e4a8ef4549cc12300e0eb743"
server_key_version: "encryptionkey@0"
I20260927 04:59:21.731581 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "d8521f8295d3486995ada4641bc581ca"
format_stamp: "Formatted at 2026-09-27 04:59:21 on dist-test-slave-39f2"
server_key: "fc662825f1bd5f9f79bc06fc4aa083b4"
server_key_iv: "6044b3e2e4a8ef4549cc12300e0eb743"
server_key_version: "encryptionkey@0"
I20260927 04:59:21.736474 24898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 04:59:21.740360 24913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:21.741662 24898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:21.741843 24898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "d8521f8295d3486995ada4641bc581ca"
format_stamp: "Formatted at 2026-09-27 04:59:21 on dist-test-slave-39f2"
server_key: "fc662825f1bd5f9f79bc06fc4aa083b4"
server_key_iv: "6044b3e2e4a8ef4549cc12300e0eb743"
server_key_version: "encryptionkey@0"
I20260927 04:59:21.742069 24898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:21.753980 24898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:21.758086 24898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:21.758392 24898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:21.759392 24898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:21.760730 24898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:21.760855 24898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:21.761004 24898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:21.761076 24898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:21.794064 24898 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:37375
I20260927 04:59:21.794144 25026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:37375 every 8 connection(s)
I20260927 04:59:21.795364 24898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:21.804723 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 24898
I20260927 04:59:21.804967 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:21.805629 21157 external_mini_cluster.cc:1428] Setting key d64c020fdb9775b553962cd6608aa99e
Sep 27 04:59:21 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485161, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:21.816174 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:21.798117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:35081 (local address 127.20.169.126:33281)
0927 04:59:21.798627 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:21.798632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:21.799530 (+ 898us) server_negotiation.cc:407] Connection header received
0927 04:59:21.801422 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:21.801427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:21.801496 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:21.801595 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:21.803122 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:21.803862 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:21.805821 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:21.806266 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:21.806428 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:21.810198 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:21.810204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:21.810207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:21.810240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:21.812863 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:21.813604 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:21.813610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:21.813613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:21.813735 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:21.814168 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:21.814176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:21.814182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:21.814557 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:21.814709 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:21.815442 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 04:59:21.815694 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":308,"threads_started":1}
I20260927 04:59:21.816907 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:21.798606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33281 (local address 127.20.169.65:35081)
0927 04:59:21.799311 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:21.799393 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:21.801115 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:21.801829 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:21.801838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:21.802352 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:21.802949 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:21.802967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:21.804040 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:21.804045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:21.805614 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:21.805634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:21.806520 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:21.806530 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:21.806815 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:21.807572 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:21.807597 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:21.809935 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:21.813011 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:21.813020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:21.813039 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:21.813475 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:21.813849 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:21.813854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:21.813858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:21.814045 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:21.814692 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:21.814699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:21.815036 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 04:59:21.815263 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 04:59:21.815722 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":294,"threads_started":1}
I20260927 04:59:21.818542 25027 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33281
I20260927 04:59:21.818885 25027 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:21.819777 25027 heartbeater.cc:511] Master 127.20.169.126:33281 requested a full tablet report, sending...
I20260927 04:59:21.822124 24836 ts_manager.cc:194] Registered new tserver with Master: d8521f8295d3486995ada4641bc581ca (127.20.169.65:37375)
I20260927 04:59:21.823570 24836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:35081
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:21.851576 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33281
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:22.173096 25033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:22.173372 25033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:22.173527 25033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:22.180702 25033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:22.180819 25033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:22.181005 25033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:22.190837 25033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:22.192070 25033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:22.193403 25033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:22.206704 25041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:22.206753 25039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:22.206668 25038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:22.207837 25033 server_base.cc:1034] running on GCE node
I20260927 04:59:22.208428 25033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:22.209404 25033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:22.210630 25033 hybrid_clock.cc:648] HybridClock initialized: now 1790485162210574 us; error 48 us; skew 500 ppm
I20260927 04:59:22.210944 25033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:22.214012 25033 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:22.215401 25033 webserver.cc:460] Webserver started at http://127.20.169.66:46271/ using document root <none> and password file <none>
I20260927 04:59:22.216024 25033 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:22.216118 25033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:22.216380 25033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:22.219259 25033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "34fad3cdc1564392a5cc9c2a3a9f48d3"
format_stamp: "Formatted at 2026-09-27 04:59:22 on dist-test-slave-39f2"
server_key: "f2a51cc54d2004af415c4a277f4960f8"
server_key_iv: "2f1ee51f2e2aee5a12203b98212c8879"
server_key_version: "encryptionkey@0"
I20260927 04:59:22.219974 25033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "34fad3cdc1564392a5cc9c2a3a9f48d3"
format_stamp: "Formatted at 2026-09-27 04:59:22 on dist-test-slave-39f2"
server_key: "f2a51cc54d2004af415c4a277f4960f8"
server_key_iv: "2f1ee51f2e2aee5a12203b98212c8879"
server_key_version: "encryptionkey@0"
I20260927 04:59:22.224792 25033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 04:59:22.228047 25048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:22.229415 25033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:22.229566 25033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "34fad3cdc1564392a5cc9c2a3a9f48d3"
format_stamp: "Formatted at 2026-09-27 04:59:22 on dist-test-slave-39f2"
server_key: "f2a51cc54d2004af415c4a277f4960f8"
server_key_iv: "2f1ee51f2e2aee5a12203b98212c8879"
server_key_version: "encryptionkey@0"
I20260927 04:59:22.229737 25033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:22.247255 25033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:22.251572 25033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:22.251871 25033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:22.252882 25033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:22.254333 25033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:22.254405 25033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:22.254537 25033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:22.254604 25033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:22.287864 25033 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45211
I20260927 04:59:22.287909 25161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45211 every 8 connection(s)
I20260927 04:59:22.289218 25033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:22.290189 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25033
I20260927 04:59:22.290366 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:22.290839 21157 external_mini_cluster.cc:1428] Setting key d88f36ef670a2e856b76600d55634ad2
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:59:22.311691 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.292171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:46019 (local address 127.20.169.126:33281)
0927 04:59:22.292382 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:22.292389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:22.293704 (+ 1315us) server_negotiation.cc:407] Connection header received
0927 04:59:22.295481 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:22.295489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:22.295579 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:22.295697 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:22.297296 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.298402 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.299962 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.300747 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.300921 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.304597 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:22.304606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:22.304614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:22.304662 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:22.308524 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.309276 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.309283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.309288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.309449 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.309846 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.309852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.309856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.310264 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:22.310427 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:22.311047 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 04:59:22.311308 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 04:59:22.312319 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.292633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33281 (local address 127.20.169.66:46019)
0927 04:59:22.293531 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:22.293583 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:22.295210 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:22.295905 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:22.295913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:22.296468 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:22.297130 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:22.297150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.298561 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.298564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:22.299812 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:22.299824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.301007 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.301012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:22.301191 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.301905 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:22.301932 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:22.304367 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:22.308661 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:22.308667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:22.308678 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:22.309163 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:22.309562 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:22.309565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:22.309567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:22.309725 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:22.310397 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:22.310402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:22.310687 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 04:59:22.310893 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 04:59:22.311243 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":352,"threads_started":1}
I20260927 04:59:22.314042 25162 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33281
I20260927 04:59:22.314347 25162 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 04:59:22.315253 25162 heartbeater.cc:511] Master 127.20.169.126:33281 requested a full tablet report, sending...
I20260927 04:59:22.316967 24836 ts_manager.cc:194] Registered new tserver with Master: 34fad3cdc1564392a5cc9c2a3a9f48d3 (127.20.169.66:45211)
I20260927 04:59:22.318060 24836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:46019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:22.340729 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33281
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:22.670478 25167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:22.670735 25167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:22.670898 25167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:22.678169 25167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:22.678269 25167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:22.678503 25167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:22.689396 25167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:22.690764 25167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:22.692191 25167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:22.705933 25173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:22.705968 25172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:22.705943 25175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:22.706888 25167 server_base.cc:1034] running on GCE node
I20260927 04:59:22.707502 25167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:22.708455 25167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:22.709677 25167 hybrid_clock.cc:648] HybridClock initialized: now 1790485162709627 us; error 45 us; skew 500 ppm
I20260927 04:59:22.709983 25167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:22.713265 25167 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:22.714808 25167 webserver.cc:460] Webserver started at http://127.20.169.67:41087/ using document root <none> and password file <none>
I20260927 04:59:22.715490 25167 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:22.715680 25167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:22.715979 25167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:22.718945 25167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "dd98c8c5e45c4288a3630a6c9d65bb15"
format_stamp: "Formatted at 2026-09-27 04:59:22 on dist-test-slave-39f2"
server_key: "30673935d16348806095488827a85041"
server_key_iv: "f94783e54419705e6c53f84bd57abcb8"
server_key_version: "encryptionkey@0"
I20260927 04:59:22.719691 25167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "dd98c8c5e45c4288a3630a6c9d65bb15"
format_stamp: "Formatted at 2026-09-27 04:59:22 on dist-test-slave-39f2"
server_key: "30673935d16348806095488827a85041"
server_key_iv: "f94783e54419705e6c53f84bd57abcb8"
server_key_version: "encryptionkey@0"
I20260927 04:59:22.724635 25167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 04:59:22.728225 25182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:22.729561 25167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:22.729724 25167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "dd98c8c5e45c4288a3630a6c9d65bb15"
format_stamp: "Formatted at 2026-09-27 04:59:22 on dist-test-slave-39f2"
server_key: "30673935d16348806095488827a85041"
server_key_iv: "f94783e54419705e6c53f84bd57abcb8"
server_key_version: "encryptionkey@0"
I20260927 04:59:22.729911 25167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:22.748134 25167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:22.752345 25167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:22.752629 25167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:22.753633 25167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:22.754951 25167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:22.755052 25167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:22.755203 25167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:22.755278 25167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:22.788282 25167 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:39391
I20260927 04:59:22.788367 25295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:39391 every 8 connection(s)
I20260927 04:59:22.789659 25167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:22.799230 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25167
I20260927 04:59:22.799396 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:22.799925 21157 external_mini_cluster.cc:1428] Setting key 1a4d131ffb4962aa4abf62a20d827a6b
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:22.810788 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.792310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:36593 (local address 127.20.169.126:33281)
0927 04:59:22.792525 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:22.792530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:22.793889 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 04:59:22.796436 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:22.796440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:22.796509 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:22.796588 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:22.798525 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.799637 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.801558 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.802146 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.802272 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.805620 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:22.805625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:22.805628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:22.805659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:22.808211 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.808895 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.808900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.808902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.809024 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.809386 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.809389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.809390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.809636 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:22.809743 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:22.810370 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 04:59:22.810543 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 04:59:22.811724 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.792796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33281 (local address 127.20.169.67:36593)
0927 04:59:22.793681 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:22.793741 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:22.796119 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:22.796882 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:22.796893 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:22.797622 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:22.798344 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:22.798366 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.799802 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.799815 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:22.801417 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:22.801432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.802356 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.802362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:22.802556 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.803301 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:22.803336 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:22.805423 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:22.808334 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:22.808340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:22.808351 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:22.808739 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:22.809117 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:22.809120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:22.809123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:22.809287 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:22.809738 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:22.809743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:22.810046 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 04:59:22.810278 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 04:59:22.810644 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":306,"threads_started":1}
I20260927 04:59:22.813177 25296 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33281
I20260927 04:59:22.813522 25296 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:22.814436 25296 heartbeater.cc:511] Master 127.20.169.126:33281 requested a full tablet report, sending...
I20260927 04:59:22.816216 24836 ts_manager.cc:194] Registered new tserver with Master: dd98c8c5e45c4288a3630a6c9d65bb15 (127.20.169.67:39391)
I20260927 04:59:22.817122 24836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:36593
I20260927 04:59:22.827045 25027 heartbeater.cc:503] Master 127.20.169.126:33281 was elected leader, sending a full tablet report...
I20260927 04:59:22.828385 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:22.877413 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.862292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.20.169.65:37375)
0927 04:59:22.862974 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:22.862982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:22.863008 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:59:22.863076 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:22.863082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:22.863289 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:22.863439 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:22.864781 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.865866 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.867287 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.867870 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.867968 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.871779 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:22.871790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:22.871803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:22.871867 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:22.874782 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.875411 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.875418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.875420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.875535 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.875947 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.875950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.875952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.876281 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:22.876497 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:22.876800 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 04:59:22.877058 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":286,"threads_started":1}
Sep 27 04:59:22 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:22.895548 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.881938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.20.169.126:33281)
0927 04:59:22.882168 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:22.882173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:22.882193 (+ 20us) server_negotiation.cc:407] Connection header received
0927 04:59:22.882345 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:22.882350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:22.882417 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:22.882522 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:22.883773 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.884501 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.885798 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.886236 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.886346 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.890102 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:22.890107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:22.890110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:22.890143 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:22.892698 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.893421 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.893426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.893428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.893597 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.894109 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.894113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.894115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.894464 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:22.894651 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:22.895000 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 04:59:22.895182 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 04:59:22.896215 24835 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42554
I20260927 04:59:22.897444 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24898
I20260927 04:59:22.909642 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:22.898411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.20.169.65:37375)
0927 04:59:22.898643 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:22.898650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:22.898728 (+ 78us) server_negotiation.cc:407] Connection header received
0927 04:59:22.898852 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:22.898856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:22.898923 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:22.899011 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:22.900107 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.900931 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.902089 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:22.902514 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:22.902615 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:22.904651 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:22.904655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:22.904658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:22.904687 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:22.907343 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.907911 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.907915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.907918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.908038 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:22.908385 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:22.908390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:22.908393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:22.908668 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:22.908931 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:22.909132 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 04:59:22.909296 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 04:59:22.910481 25022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:22.994656 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24898
I20260927 04:59:23.011916 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25033
Sep 27 04:59:23 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:23.029524 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:23.012918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.20.169.66:45211)
0927 04:59:23.013585 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:23.013592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:23.013618 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:59:23.013687 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:23.013695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:23.013916 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:23.014050 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:23.015357 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:23.016513 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:23.018076 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:23.018651 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:23.018787 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:23.023291 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:23.023306 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:23.023322 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:23.023366 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:23.026743 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:23.027442 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:23.027450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:23.027453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:23.027594 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:23.027983 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:23.027986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:23.027988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:23.028350 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:23.028582 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:23.028989 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 04:59:23.029196 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":308,"threads_started":1}
I20260927 04:59:23.030903 25157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:23.114043 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25033
I20260927 04:59:23.131276 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25167
Sep 27 04:59:23 dist-test-slave-39f2 krb5kdc[24790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:23.148103 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:23.132374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.20.169.67:39391)
0927 04:59:23.133058 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:23.133066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:23.133096 (+ 30us) server_negotiation.cc:407] Connection header received
0927 04:59:23.133158 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:23.133165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:23.133384 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:23.133560 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:23.134845 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:23.135929 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:23.137342 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:23.137907 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:23.138053 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:23.142243 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:23.142253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:23.142267 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:23.142310 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:23.145225 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:23.145913 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:23.145921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:23.145923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:23.146050 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:23.146482 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:23.146488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:23.146491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:23.146840 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:23.147160 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:23.147492 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 04:59:23.147754 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":320,"threads_started":1}
I20260927 04:59:23.149796 25291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:23.233500 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25167
I20260927 04:59:23.250823 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 24803
I20260927 04:59:23.252100 24866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:23.355876 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 24803
2026-09-27T04:59:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3920 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:23 dist-test-slave-39f2 krb5kdc[25315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:23 dist-test-slave-39f2 krb5kdc[25315](info): set up 2 sockets
Sep 27 04:59:23 dist-test-slave-39f2 krb5kdc[25315](info): commencing operation
krb5kdc: starting...
W20260927 04:59:24.456962 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:24 dist-test-slave-39f2 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:24.664650 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45821
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:24.980880 25328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:24.981190 25328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:24.981367 25328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:24.988317 25328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:24.988407 25328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:24.988452 25328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:24.988508 25328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:24.988549 25328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:24.998039 25328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:24.999183 25328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:25.000409 25328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:25.011720 25333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:25.011833 25334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:25.012516 25336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:25.013301 25328 server_base.cc:1034] running on GCE node
I20260927 04:59:25.014124 25328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:25.015538 25328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:25.016980 25328 hybrid_clock.cc:648] HybridClock initialized: now 1790485165016895 us; error 73 us; skew 500 ppm
I20260927 04:59:25.017417 25328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:25 dist-test-slave-39f2 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485165, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:25.020761 25328 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:25.022058 25328 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:25.022245 25328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:25.022606 25328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:25.025601 25328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "327a3fc300454d01a614ccec33b2420f"
format_stamp: "Formatted at 2026-09-27 04:59:25 on dist-test-slave-39f2"
server_key: "8c13a1b5c0cde641631c7e9e2ccfc7d5"
server_key_iv: "330dc6c17c5cfa93955a6e7636f74754"
server_key_version: "encryptionkey@0"
I20260927 04:59:25.026401 25328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "327a3fc300454d01a614ccec33b2420f"
format_stamp: "Formatted at 2026-09-27 04:59:25 on dist-test-slave-39f2"
server_key: "8c13a1b5c0cde641631c7e9e2ccfc7d5"
server_key_iv: "330dc6c17c5cfa93955a6e7636f74754"
server_key_version: "encryptionkey@0"
I20260927 04:59:25.031386 25328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:59:25.034901 25342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:25.036207 25328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 04:59:25.036458 25328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "327a3fc300454d01a614ccec33b2420f"
format_stamp: "Formatted at 2026-09-27 04:59:25 on dist-test-slave-39f2"
server_key: "8c13a1b5c0cde641631c7e9e2ccfc7d5"
server_key_iv: "330dc6c17c5cfa93955a6e7636f74754"
server_key_version: "encryptionkey@0"
I20260927 04:59:25.036669 25328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:25.056638 25328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:25.061172 25328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:25.061493 25328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:25.083508 25328 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45821
I20260927 04:59:25.083478 25393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45821 every 8 connection(s)
I20260927 04:59:25.084553 25328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:25.088384 25394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:25.093549 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25328
I20260927 04:59:25.093792 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:25.094571 21157 external_mini_cluster.cc:1428] Setting key a6398b9feae7cc6b493654b406e5edff
I20260927 04:59:25.099032 25394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f: Bootstrap starting.
I20260927 04:59:25.102715 25394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:25.103796 25394 log.cc:824] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:25.106454 25394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f: No bootstrap required, opened a new log
Sep 27 04:59:25 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:25.114068 25394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327a3fc300454d01a614ccec33b2420f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45821 } }
I20260927 04:59:25.114460 25394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:25.114604 25394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327a3fc300454d01a614ccec33b2420f, State: Initialized, Role: FOLLOWER
I20260927 04:59:25.115118 25394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327a3fc300454d01a614ccec33b2420f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45821 } }
I20260927 04:59:25.115404 25394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:25.115548 25394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:25.115717 25394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:25.118005 25394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327a3fc300454d01a614ccec33b2420f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45821 } }
I20260927 04:59:25.118505 25394 leader_election.cc:302] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327a3fc300454d01a614ccec33b2420f; no voters:
I20260927 04:59:25.119227 25394 leader_election.cc:288] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:25.119345 25399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:25.120415 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:25.096949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.20.169.126:45821)
0927 04:59:25.097840 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:25.097853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:25.097901 (+ 48us) server_negotiation.cc:407] Connection header received
0927 04:59:25.099979 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:25.100004 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:25.100425 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:25.100865 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:25.102729 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:25.104447 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:25.105923 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:25.106744 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:25.106977 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:25.111202 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:25.111210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:25.111228 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:25.111269 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:25.115113 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:25.116048 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:25.116058 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:25.116067 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:25.116310 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:25.116831 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:25.116840 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:25.116844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:25.117502 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:25.117862 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:25.118410 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 04:59:25.118884 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":412,"threads_started":1}
I20260927 04:59:25.121104 25399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [term 1 LEADER]: Becoming Leader. State: Replica: 327a3fc300454d01a614ccec33b2420f, State: Running, Role: LEADER
I20260927 04:59:25.121765 25394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:25.121685 25399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327a3fc300454d01a614ccec33b2420f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45821 } }
I20260927 04:59:25.126742 25401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327a3fc300454d01a614ccec33b2420f. Latest consensus state: current_term: 1 leader_uuid: "327a3fc300454d01a614ccec33b2420f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327a3fc300454d01a614ccec33b2420f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45821 } } }
I20260927 04:59:25.126705 25400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "327a3fc300454d01a614ccec33b2420f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327a3fc300454d01a614ccec33b2420f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45821 } } }
I20260927 04:59:25.127179 25401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:25.127183 25400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:25.128857 25410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:25.135417 25410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:25.145486 25410 catalog_manager.cc:1347] Generated new cluster ID: f9fc574e35834fbabe3ba22ca2604c6e
I20260927 04:59:25.145606 25410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:25.155602 25410 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:25.156945 25410 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:25.170703 25410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f: Generated new TSK 0
I20260927 04:59:25.171605 25410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:25.229146 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45821
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:25.563946 25420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:25.564266 25420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:25.564443 25420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:25.572014 25420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:25.572158 25420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:25.572360 25420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:25.582024 25420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:25.583328 25420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:25.584613 25420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:25.596989 25428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:25.597113 25426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:25.597291 25425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:25.598162 25420 server_base.cc:1034] running on GCE node
I20260927 04:59:25.598757 25420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:25.599743 25420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:25.600978 25420 hybrid_clock.cc:648] HybridClock initialized: now 1790485165600936 us; error 67 us; skew 500 ppm
I20260927 04:59:25.601392 25420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:25 dist-test-slave-39f2 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485165, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:25.604664 25420 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:25.605964 25420 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:25.606101 25420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:25.606410 25420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:25.609432 25420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "1c6ba070448a42019a52c579cdf7c031"
format_stamp: "Formatted at 2026-09-27 04:59:25 on dist-test-slave-39f2"
server_key: "d7ace0699587cbd163297fbdb5ee7b01"
server_key_iv: "85d4e6da2c6aa78560179ae7bdbd81d1"
server_key_version: "encryptionkey@0"
I20260927 04:59:25.610245 25420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "1c6ba070448a42019a52c579cdf7c031"
format_stamp: "Formatted at 2026-09-27 04:59:25 on dist-test-slave-39f2"
server_key: "d7ace0699587cbd163297fbdb5ee7b01"
server_key_iv: "85d4e6da2c6aa78560179ae7bdbd81d1"
server_key_version: "encryptionkey@0"
I20260927 04:59:25.615216 25420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 04:59:25.618618 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:25.619968 25420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:25.620101 25420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "1c6ba070448a42019a52c579cdf7c031"
format_stamp: "Formatted at 2026-09-27 04:59:25 on dist-test-slave-39f2"
server_key: "d7ace0699587cbd163297fbdb5ee7b01"
server_key_iv: "85d4e6da2c6aa78560179ae7bdbd81d1"
server_key_version: "encryptionkey@0"
I20260927 04:59:25.620273 25420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:25.632576 25420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:25.636473 25420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:25.636696 25420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:25.637677 25420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:25.638967 25420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:25.639048 25420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:25.639125 25420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:25.639178 25420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:25.672104 25420 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:44329
I20260927 04:59:25.672149 25548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:44329 every 8 connection(s)
I20260927 04:59:25.673246 25420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:25.682148 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25420
I20260927 04:59:25.682344 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:25.682971 21157 external_mini_cluster.cc:1428] Setting key fd86ca43bfade1fb490355979fc4512b
Sep 27 04:59:25 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485165, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:25.693215 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:25.676043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:40951 (local address 127.20.169.126:45821)
0927 04:59:25.676448 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:25.676453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:25.677565 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 04:59:25.679400 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:25.679405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:25.679488 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:25.679616 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:25.681204 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:25.681938 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:25.683636 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:25.684135 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:25.684282 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:25.687654 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:25.687666 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:25.687669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:25.687700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:25.690256 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:25.690945 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:25.690950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:25.690956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:25.691073 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:25.691504 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:25.691509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:25.691512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:25.691765 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:25.691928 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:25.692595 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 04:59:25.692847 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":200,"threads_started":1}
I20260927 04:59:25.694166 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:25.676626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45821 (local address 127.20.169.65:40951)
0927 04:59:25.677406 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:25.677444 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:25.679119 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:25.679851 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:25.679860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:25.680367 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:25.681041 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:25.681059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:25.682100 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:25.682110 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:25.683485 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:25.683496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:25.684387 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:25.684397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:25.684624 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:25.685375 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:25.685402 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:25.687446 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:25.690395 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:25.690404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:25.690420 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:25.690823 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:25.691181 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:25.691187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:25.691191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:25.691386 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:25.691899 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:25.691907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:25.692222 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 04:59:25.692493 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 04:59:25.692995 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":322,"threads_started":1}
I20260927 04:59:25.695761 25549 heartbeater.cc:347] Connected to a master server at 127.20.169.126:45821
I20260927 04:59:25.696070 25549 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:25.696698 25549 heartbeater.cc:511] Master 127.20.169.126:45821 requested a full tablet report, sending...
I20260927 04:59:25.698788 25359 ts_manager.cc:194] Registered new tserver with Master: 1c6ba070448a42019a52c579cdf7c031 (127.20.169.65:44329)
I20260927 04:59:25.700181 25359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:40951
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:25.728868 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45821
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:26.053722 25555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:26.054016 25555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:26.054113 25555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:26.061264 25555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:26.061376 25555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:26.061491 25555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:26.071408 25555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:26.072692 25555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:26.073952 25555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:26.086076 25560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:26.086201 25561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:26.087064 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:26.087308 25555 server_base.cc:1034] running on GCE node
I20260927 04:59:26.087947 25555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:26.089075 25555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:26.090305 25555 hybrid_clock.cc:648] HybridClock initialized: now 1790485166090227 us; error 86 us; skew 500 ppm
I20260927 04:59:26.090631 25555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:26 dist-test-slave-39f2 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485166, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:26.094218 25555 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:26.097519 25555 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:26.097653 25555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:26.097972 25555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:26.101001 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "68dff50e5965426398790460b24ff78b"
format_stamp: "Formatted at 2026-09-27 04:59:26 on dist-test-slave-39f2"
server_key: "eabf062c276d0b8c60fb89d6d03612c7"
server_key_iv: "0b9ef7405645e794c5a49c032412643e"
server_key_version: "encryptionkey@0"
I20260927 04:59:26.101802 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "68dff50e5965426398790460b24ff78b"
format_stamp: "Formatted at 2026-09-27 04:59:26 on dist-test-slave-39f2"
server_key: "eabf062c276d0b8c60fb89d6d03612c7"
server_key_iv: "0b9ef7405645e794c5a49c032412643e"
server_key_version: "encryptionkey@0"
I20260927 04:59:26.106810 25555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:59:26.110157 25569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:26.111434 25555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:26.111617 25555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "68dff50e5965426398790460b24ff78b"
format_stamp: "Formatted at 2026-09-27 04:59:26 on dist-test-slave-39f2"
server_key: "eabf062c276d0b8c60fb89d6d03612c7"
server_key_iv: "0b9ef7405645e794c5a49c032412643e"
server_key_version: "encryptionkey@0"
I20260927 04:59:26.111848 25555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:26.134243 25555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:26.138549 25555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:26.138844 25555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:26.139806 25555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:26.141294 25555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:26.141422 25555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:26.141559 25555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:26.141651 25555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:26.175300 25555 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:46213
I20260927 04:59:26.175397 25682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:46213 every 8 connection(s)
I20260927 04:59:26.176173 25555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:26.178347 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25555
I20260927 04:59:26.178529 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:26.179144 21157 external_mini_cluster.cc:1428] Setting key c0952c060d4721a64ad1a3fcfa1c38ed
Sep 27 04:59:26 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485166, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:26.196287 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.178854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:41063 (local address 127.20.169.126:45821)
0927 04:59:26.179046 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:26.179050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:26.180580 (+ 1530us) server_negotiation.cc:407] Connection header received
0927 04:59:26.182434 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:26.182440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:26.182524 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:26.182636 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:26.184165 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.184958 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.186485 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.186947 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.187076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.190587 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:26.190592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:26.190595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:26.190626 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:26.193312 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:26.194008 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:26.194015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:26.194018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:26.194154 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:26.194575 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:26.194580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:26.194584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:26.194891 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:26.195063 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:26.195718 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 04:59:26.195950 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 04:59:26.197325 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.179533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45821 (local address 127.20.169.66:41063)
0927 04:59:26.180406 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:26.180452 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:26.182173 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:26.182890 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:26.182898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:26.183402 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:26.183988 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.184009 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.185097 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.185107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.186322 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.186344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.187172 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.187179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.187400 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.188103 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:26.188124 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:26.190315 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:26.193454 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:26.193463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:26.193479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:26.193886 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:26.194265 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:26.194270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:26.194274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:26.194460 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:26.195021 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:26.195029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:26.195370 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 04:59:26.195628 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 04:59:26.196149 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":385,"threads_started":1}
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:59:26.198948 25683 heartbeater.cc:347] Connected to a master server at 127.20.169.126:45821
I20260927 04:59:26.199254 25683 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:26.199970 25683 heartbeater.cc:511] Master 127.20.169.126:45821 requested a full tablet report, sending...
I20260927 04:59:26.201404 25359 ts_manager.cc:194] Registered new tserver with Master: 68dff50e5965426398790460b24ff78b (127.20.169.66:46213)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 04:59:26.202270 25359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:41063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:26.226280 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45821
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:26.555361 25688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:26.555639 25688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:26.555742 25688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:26.563261 25688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:26.563406 25688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:26.563540 25688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:26.573796 25688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:45821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:26.575187 25688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:26.576612 25688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:26.589942 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:26.590001 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:26.590616 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:26.590974 25688 server_base.cc:1034] running on GCE node
I20260927 04:59:26.591528 25688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:26.592442 25688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:26.593672 25688 hybrid_clock.cc:648] HybridClock initialized: now 1790485166593630 us; error 54 us; skew 500 ppm
I20260927 04:59:26.593994 25688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:26 dist-test-slave-39f2 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485166, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:26.597146 25688 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:26.598598 25688 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:26.598753 25688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:26.599082 25688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:26.601974 25688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "b173d7af4931431a831a3dccc1c501f4"
format_stamp: "Formatted at 2026-09-27 04:59:26 on dist-test-slave-39f2"
server_key: "c9ff435251dcdcde40256b7015626012"
server_key_iv: "9a132948ef324c52893c95d0d93a3201"
server_key_version: "encryptionkey@0"
I20260927 04:59:26.602792 25688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "b173d7af4931431a831a3dccc1c501f4"
format_stamp: "Formatted at 2026-09-27 04:59:26 on dist-test-slave-39f2"
server_key: "c9ff435251dcdcde40256b7015626012"
server_key_iv: "9a132948ef324c52893c95d0d93a3201"
server_key_version: "encryptionkey@0"
I20260927 04:59:26.607667 25688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 04:59:26.611181 25703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:26.612517 25688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 04:59:26.612689 25688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "b173d7af4931431a831a3dccc1c501f4"
format_stamp: "Formatted at 2026-09-27 04:59:26 on dist-test-slave-39f2"
server_key: "c9ff435251dcdcde40256b7015626012"
server_key_iv: "9a132948ef324c52893c95d0d93a3201"
server_key_version: "encryptionkey@0"
I20260927 04:59:26.612974 25688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:26.634706 25688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:26.638547 25688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:26.638814 25688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:26.639804 25688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:26.641228 25688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:26.641353 25688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:26.641496 25688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:26.641585 25688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:26.676134 25688 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:46123
I20260927 04:59:26.676193 25815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:46123 every 8 connection(s)
I20260927 04:59:26.677181 25688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:26.686215 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25688
I20260927 04:59:26.686414 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:26.686856 21157 external_mini_cluster.cc:1428] Setting key e3d569787bf6f6f46a0f415a3f484a38
Sep 27 04:59:26 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485166, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:26.697609 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.679896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:33817 (local address 127.20.169.126:45821)
0927 04:59:26.680097 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:26.680102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:26.681232 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 04:59:26.683175 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:26.683181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:26.683306 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:26.683437 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:26.685060 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.685975 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.687910 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.688432 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.688566 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.692115 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:26.692121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:26.692124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:26.692155 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:26.694587 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:26.695356 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:26.695360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:26.695363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:26.695505 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:26.695895 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:26.695899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:26.695901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:26.696192 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:26.696343 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:26.697126 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 04:59:26.697309 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:59:26.698658 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.680239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45821 (local address 127.20.169.67:33817)
0927 04:59:26.681066 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:26.681103 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:26.682902 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:26.683679 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:26.683687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:26.684236 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:26.684874 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.684896 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.686177 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.686181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.687747 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.687758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.688663 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.688670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.688966 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.689709 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:26.689739 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:26.691917 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:26.694718 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:26.694725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:26.694738 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:26.695184 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:26.695612 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:26.695615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:26.695618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:26.695783 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:26.696343 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:26.696349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:26.696661 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 04:59:26.696994 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 04:59:26.697407 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":269,"threads_started":1}
I20260927 04:59:26.700135 25816 heartbeater.cc:347] Connected to a master server at 127.20.169.126:45821
I20260927 04:59:26.700471 25816 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:26.701256 25816 heartbeater.cc:511] Master 127.20.169.126:45821 requested a full tablet report, sending...
I20260927 04:59:26.702839 25359 ts_manager.cc:194] Registered new tserver with Master: b173d7af4931431a831a3dccc1c501f4 (127.20.169.67:46123)
I20260927 04:59:26.703154 25549 heartbeater.cc:503] Master 127.20.169.126:45821 was elected leader, sending a full tablet report...
I20260927 04:59:26.703872 25359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:33817
I20260927 04:59:26.714866 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:59:26.731839 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.719504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.20.169.126:45821)
0927 04:59:26.719699 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:26.719703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:26.720136 (+ 433us) server_negotiation.cc:407] Connection header received
0927 04:59:26.720274 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:26.720277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:26.720338 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:26.720419 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:26.721947 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.722723 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.724064 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.724559 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.724655 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.726897 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:26.726902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:26.726905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:26.726940 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:26.729537 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:26.730154 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:26.730158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:26.730160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:26.730281 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:26.730667 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:26.730670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:26.730672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:26.730974 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:26.731152 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:26.731454 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 04:59:26.731602 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 04:59:26.736083 25358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:26.743759 25358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:26.787711 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.756631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.20.169.67:46123)
0927 04:59:26.778717 (+ 22086us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:26.778728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:26.778773 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:59:26.778898 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:26.778906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:26.779204 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:26.779443 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:26.780354 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.782079 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.785347 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.786515 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.786650 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.786889 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:26.787180 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 04:59:26.787368 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21890,"thread_start_us":21721,"threads_started":1}
I20260927 04:59:26.787814 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.756388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:46123 (local address 127.0.0.1:53910)
0927 04:59:26.768209 (+ 11821us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:26.768269 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:26.768496 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:26.779465 (+ 10969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:26.779481 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:26.779545 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:26.780034 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.780057 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.782249 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.782257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.785168 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.785187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.786697 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.786704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.786902 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.786966 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 04:59:26.787128 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 04:59:26.787329 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11457,"mutex_wait_us":62,"thread_start_us":23557,"threads_started":1}
I20260927 04:59:26.788949 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.756573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:44329 (local address 127.0.0.1:59116)
0927 04:59:26.779511 (+ 22938us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:26.779534 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:26.779654 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:26.780641 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:26.780645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:26.780670 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:26.781165 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.781178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.782855 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.782863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.785480 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.785493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.788049 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.788060 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.788202 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.788227 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:59:26.788334 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:59:26.788491 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22792,"mutex_wait_us":62,"thread_start_us":11137,"threads_started":1}
I20260927 04:59:26.789320 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.757252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.20.169.65:44329)
0927 04:59:26.780010 (+ 22758us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:26.780019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:26.780050 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:59:26.780149 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:26.780160 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:26.780422 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:26.780647 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:26.781436 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.782727 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.785654 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.787261 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.787405 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.787644 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:26.788724 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0927 04:59:26.788949 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22504,"thread_start_us":22455,"threads_started":1}
I20260927 04:59:26.789635 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.757145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:46213 (local address 127.0.0.1:39840)
0927 04:59:26.780207 (+ 23062us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:26.780225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:26.780786 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:26.781483 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:26.781489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:26.781517 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:26.781880 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.781892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.783930 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.783939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.786397 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:26.786414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.788775 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.788781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:26.788991 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.789020 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 04:59:26.789110 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 04:59:26.789228 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22911,"mutex_wait_us":77,"thread_start_us":22049,"threads_started":1}
I20260927 04:59:26.790335 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:26.757796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.20.169.66:46213)
0927 04:59:26.779012 (+ 21216us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:26.779020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:26.780337 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 04:59:26.781067 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:26.781075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:26.781297 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:26.781490 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:26.782077 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.783742 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.786571 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:26.788314 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:26.788452 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:26.788638 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:26.789845 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0927 04:59:26.790005 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21071,"thread_start_us":20928,"threads_started":1}
I20260927 04:59:26.792214 25751 tablet_service.cc:1467] Processing CreateTablet for tablet 0d941c3932ce4eab98b3d6ecae007b4b (DEFAULT_TABLE table=test-table [id=e323e5026675484fa8fd750f09f38040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:26.793596 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d941c3932ce4eab98b3d6ecae007b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:26.793443 25482 tablet_service.cc:1467] Processing CreateTablet for tablet 0d941c3932ce4eab98b3d6ecae007b4b (DEFAULT_TABLE table=test-table [id=e323e5026675484fa8fd750f09f38040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:26.794201 25618 tablet_service.cc:1467] Processing CreateTablet for tablet 0d941c3932ce4eab98b3d6ecae007b4b (DEFAULT_TABLE table=test-table [id=e323e5026675484fa8fd750f09f38040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:26.795068 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d941c3932ce4eab98b3d6ecae007b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:26.795632 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d941c3932ce4eab98b3d6ecae007b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:26.806164 25834 tablet_bootstrap.cc:492] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Bootstrap starting.
I20260927 04:59:26.809711 25834 tablet_bootstrap.cc:654] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:26.810889 25835 tablet_bootstrap.cc:492] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Bootstrap starting.
I20260927 04:59:26.811110 25834 log.cc:824] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:26.811678 25836 tablet_bootstrap.cc:492] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Bootstrap starting.
I20260927 04:59:26.815071 25836 tablet_bootstrap.cc:654] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:26.815933 25834 tablet_bootstrap.cc:492] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: No bootstrap required, opened a new log
I20260927 04:59:26.816082 25835 tablet_bootstrap.cc:654] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:26.816236 25834 ts_tablet_manager.cc:1397] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 04:59:26.816391 25836 log.cc:824] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:26.817554 25835 log.cc:824] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:26.819712 25836 tablet_bootstrap.cc:492] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: No bootstrap required, opened a new log
I20260927 04:59:26.819989 25836 ts_tablet_manager.cc:1397] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 04:59:26.820207 25835 tablet_bootstrap.cc:492] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: No bootstrap required, opened a new log
I20260927 04:59:26.820485 25835 ts_tablet_manager.cc:1397] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 04:59:26.827620 25834 raft_consensus.cc:348] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:26.828058 25834 raft_consensus.cc:374] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:26.828159 25834 raft_consensus.cc:729] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b173d7af4931431a831a3dccc1c501f4, State: Initialized, Role: FOLLOWER
I20260927 04:59:26.828783 25834 consensus_queue.cc:260] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:26.829155 25835 raft_consensus.cc:348] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:26.829619 25835 raft_consensus.cc:374] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:26.829735 25835 raft_consensus.cc:729] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68dff50e5965426398790460b24ff78b, State: Initialized, Role: FOLLOWER
I20260927 04:59:26.830176 25816 heartbeater.cc:503] Master 127.20.169.126:45821 was elected leader, sending a full tablet report...
I20260927 04:59:26.830652 25834 ts_tablet_manager.cc:1428] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 04:59:26.830741 25836 raft_consensus.cc:348] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:26.831139 25836 raft_consensus.cc:374] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:26.831214 25836 raft_consensus.cc:729] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6ba070448a42019a52c579cdf7c031, State: Initialized, Role: FOLLOWER
I20260927 04:59:26.831841 25836 consensus_queue.cc:260] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:26.831830 25835 consensus_queue.cc:260] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:26.834225 25835 ts_tablet_manager.cc:1428] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260927 04:59:26.834491 25683 heartbeater.cc:503] Master 127.20.169.126:45821 was elected leader, sending a full tablet report...
I20260927 04:59:26.834610 25836 ts_tablet_manager.cc:1428] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
W20260927 04:59:26.925714 25550 tablet.cc:2369] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:59:26.928033 25684 tablet.cc:2369] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:59:26.928896 25817 tablet.cc:2369] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:27.066550 25840 raft_consensus.cc:482] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:27.066900 25840 raft_consensus.cc:504] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:27.068696 25840 leader_election.cc:288] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c6ba070448a42019a52c579cdf7c031 (127.20.169.65:44329), 68dff50e5965426398790460b24ff78b (127.20.169.66:46213)
I20260927 04:59:27.076496 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.069218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:48985 (local address 127.20.169.66:46213)
0927 04:59:27.069532 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:27.069550 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:27.069672 (+ 122us) server_negotiation.cc:407] Connection header received
0927 04:59:27.069922 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:27.069931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:27.070061 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:27.070208 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:27.070825 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.071930 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.074262 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.075608 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.075769 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.075874 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:27.076185 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 04:59:27.076288 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 04:59:27.076764 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.069068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:46213 (local address 127.20.169.67:48985)
0927 04:59:27.069567 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:27.069588 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:27.069712 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:27.070421 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:27.070424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:27.070447 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:27.070667 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:27.070682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.072120 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.072126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:27.073998 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:27.074011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.075770 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.075776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:27.075993 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.076031 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 04:59:27.076138 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:59:27.076280 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1}
I20260927 04:59:27.077847 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.069202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:42481 (local address 127.20.169.65:44329)
0927 04:59:27.069485 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:27.069494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:27.070613 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 04:59:27.070872 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:27.070876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:27.070937 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:27.071007 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:27.071687 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.072962 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.075453 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.076790 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.077017 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.077101 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:27.077438 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 04:59:27.077578 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260927 04:59:27.077872 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.069224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:44329 (local address 127.20.169.67:42481)
0927 04:59:27.070497 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:27.070513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:27.070624 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:27.071063 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:27.071068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:27.071098 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:27.071476 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:27.071489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.073141 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.073146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:27.075259 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:27.075275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.077086 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.077092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:27.077249 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.077274 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:59:27.077395 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 04:59:27.077522 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":847,"threads_started":1}
I20260927 04:59:27.077769 25638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d941c3932ce4eab98b3d6ecae007b4b" candidate_uuid: "b173d7af4931431a831a3dccc1c501f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68dff50e5965426398790460b24ff78b" is_pre_election: true
I20260927 04:59:27.078442 25638 raft_consensus.cc:2454] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b173d7af4931431a831a3dccc1c501f4 in term 0.
I20260927 04:59:27.078851 25503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d941c3932ce4eab98b3d6ecae007b4b" candidate_uuid: "b173d7af4931431a831a3dccc1c501f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6ba070448a42019a52c579cdf7c031" is_pre_election: true
I20260927 04:59:27.079180 25707 leader_election.cc:302] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68dff50e5965426398790460b24ff78b, b173d7af4931431a831a3dccc1c501f4; no voters:
I20260927 04:59:27.079262 25503 raft_consensus.cc:2454] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b173d7af4931431a831a3dccc1c501f4 in term 0.
I20260927 04:59:27.079654 25840 raft_consensus.cc:2781] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:27.079869 25840 raft_consensus.cc:482] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:27.079991 25840 raft_consensus.cc:3037] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:27.082407 25840 raft_consensus.cc:504] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:27.083145 25840 leader_election.cc:288] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [CANDIDATE]: Term 1 election: Requested vote from peers 1c6ba070448a42019a52c579cdf7c031 (127.20.169.65:44329), 68dff50e5965426398790460b24ff78b (127.20.169.66:46213)
I20260927 04:59:27.083649 25503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d941c3932ce4eab98b3d6ecae007b4b" candidate_uuid: "b173d7af4931431a831a3dccc1c501f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6ba070448a42019a52c579cdf7c031"
I20260927 04:59:27.083716 25638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d941c3932ce4eab98b3d6ecae007b4b" candidate_uuid: "b173d7af4931431a831a3dccc1c501f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68dff50e5965426398790460b24ff78b"
I20260927 04:59:27.083853 25503 raft_consensus.cc:3037] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:27.084004 25638 raft_consensus.cc:3037] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:27.086513 25638 raft_consensus.cc:2454] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b173d7af4931431a831a3dccc1c501f4 in term 1.
I20260927 04:59:27.086933 25707 leader_election.cc:302] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68dff50e5965426398790460b24ff78b, b173d7af4931431a831a3dccc1c501f4; no voters:
I20260927 04:59:27.087141 25503 raft_consensus.cc:2454] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b173d7af4931431a831a3dccc1c501f4 in term 1.
I20260927 04:59:27.087234 25840 raft_consensus.cc:2781] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:27.088158 25840 raft_consensus.cc:686] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 1 LEADER]: Becoming Leader. State: Replica: b173d7af4931431a831a3dccc1c501f4, State: Running, Role: LEADER
I20260927 04:59:27.089048 25840 consensus_queue.cc:237] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } }
I20260927 04:59:27.094515 25359 catalog_manager.cc:5495] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b173d7af4931431a831a3dccc1c501f4 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "b173d7af4931431a831a3dccc1c501f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173d7af4931431a831a3dccc1c501f4" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 } health_report { overall_health: UNKNOWN } } }
I20260927 04:59:27.159744 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.154205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.20.169.67:46123)
0927 04:59:27.154398 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:27.154404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:27.154535 (+ 131us) server_negotiation.cc:407] Connection header received
0927 04:59:27.154788 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:27.154792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:27.154876 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:27.155043 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:27.155392 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.156090 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.157543 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.157954 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.158091 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.158584 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:27.159128 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:27.159291 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:27.159397 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 04:59:27.159480 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 04:59:27.168288 25503 raft_consensus.cc:1261] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 1 FOLLOWER]: Refusing update from remote peer b173d7af4931431a831a3dccc1c501f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:27.168334 25638 raft_consensus.cc:1261] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 1 FOLLOWER]: Refusing update from remote peer b173d7af4931431a831a3dccc1c501f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:27.169358 25845 consensus_queue.cc:1035] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c6ba070448a42019a52c579cdf7c031" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:27.169727 25840 consensus_queue.cc:1035] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68dff50e5965426398790460b24ff78b" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:27.182075 25848 mvcc.cc:205] Tried to move back new op lower bound from 7333827244710404096 to 7333827244400918528. Current Snapshot: MvccSnapshot[applied={T|T < 7333827244710404096}]
I20260927 04:59:27.189967 25849 mvcc.cc:205] Tried to move back new op lower bound from 7333827244710404096 to 7333827244400918528. Current Snapshot: MvccSnapshot[applied={T|T < 7333827244710404096}]
I20260927 04:59:27.213882 25357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:27.214201 25357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:27.218680 25357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 327a3fc300454d01a614ccec33b2420f: Sending DeleteTablet for 3 replicas of tablet 0d941c3932ce4eab98b3d6ecae007b4b
I20260927 04:59:27.219906 25751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d941c3932ce4eab98b3d6ecae007b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:27 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:53910
I20260927 04:59:27.220077 25618 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d941c3932ce4eab98b3d6ecae007b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:27 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:39840
I20260927 04:59:27.220438 25482 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d941c3932ce4eab98b3d6ecae007b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:27 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:59116
I20260927 04:59:27.223938 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25420
I20260927 04:59:27.234048 25860 tablet_replica.cc:330] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: stopping tablet replica
I20260927 04:59:27.234599 25860 raft_consensus.cc:2229] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:27.235191 25860 raft_consensus.cc:2258] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:27.237193 25859 tablet_replica.cc:330] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: stopping tablet replica
I20260927 04:59:27.237658 25859 raft_consensus.cc:2229] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:27.238022 25860 ts_tablet_manager.cc:1905] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:27.238190 25859 raft_consensus.cc:2258] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:27.240549 25859 ts_tablet_manager.cc:1905] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:27.241379 25858 tablet_replica.cc:330] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: stopping tablet replica
I20260927 04:59:27.241806 25858 raft_consensus.cc:2229] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:27.242515 25858 raft_consensus.cc:2258] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:27.246810 25860 ts_tablet_manager.cc:1918] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:27.247079 25860 log.cc:1196] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/0d941c3932ce4eab98b3d6ecae007b4b
Sep 27 04:59:27 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:27.247809 25860 ts_tablet_manager.cc:1939] T 0d941c3932ce4eab98b3d6ecae007b4b P 1c6ba070448a42019a52c579cdf7c031: Deleting consensus metadata
I20260927 04:59:27.249967 25345 catalog_manager.cc:4841] TS 1c6ba070448a42019a52c579cdf7c031 (127.20.169.65:44329): tablet 0d941c3932ce4eab98b3d6ecae007b4b (table test-table [id=e323e5026675484fa8fd750f09f38040]) successfully deleted
I20260927 04:59:27.250458 25859 ts_tablet_manager.cc:1918] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:27.250645 25859 log.cc:1196] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/0d941c3932ce4eab98b3d6ecae007b4b
I20260927 04:59:27.251298 25859 ts_tablet_manager.cc:1939] T 0d941c3932ce4eab98b3d6ecae007b4b P 68dff50e5965426398790460b24ff78b: Deleting consensus metadata
I20260927 04:59:27.253221 25346 catalog_manager.cc:4841] TS 68dff50e5965426398790460b24ff78b (127.20.169.66:46213): tablet 0d941c3932ce4eab98b3d6ecae007b4b (table test-table [id=e323e5026675484fa8fd750f09f38040]) successfully deleted
I20260927 04:59:27.255069 25858 ts_tablet_manager.cc:1905] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:27.256213 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.225261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.20.169.65:44329)
0927 04:59:27.225487 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:27.225498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:27.236455 (+ 10957us) server_negotiation.cc:407] Connection header received
0927 04:59:27.236704 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:27.236713 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:27.236902 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:27.237018 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:27.238970 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.240134 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.242250 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.242953 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.243081 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.248860 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:27.248879 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:27.248898 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:27.248944 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:27.253054 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:27.253906 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:27.253916 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:27.253919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:27.254078 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:27.254493 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:27.254499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:27.254503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:27.254863 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:27.255215 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:27.255538 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 04:59:27.255769 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 04:59:27.257004 25544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:27.264168 25858 ts_tablet_manager.cc:1918] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:27.264357 25858 log.cc:1196] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/0d941c3932ce4eab98b3d6ecae007b4b
I20260927 04:59:27.264997 25858 ts_tablet_manager.cc:1939] T 0d941c3932ce4eab98b3d6ecae007b4b P b173d7af4931431a831a3dccc1c501f4: Deleting consensus metadata
I20260927 04:59:27.267041 25344 catalog_manager.cc:4841] TS b173d7af4931431a831a3dccc1c501f4 (127.20.169.67:46123): tablet 0d941c3932ce4eab98b3d6ecae007b4b (table test-table [id=e323e5026675484fa8fd750f09f38040]) successfully deleted
I20260927 04:59:27.347846 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25420
I20260927 04:59:27.366921 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25555
Sep 27 04:59:27 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:27.384568 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.367789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.20.169.66:46213)
0927 04:59:27.368005 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:27.368012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:27.368035 (+ 23us) server_negotiation.cc:407] Connection header received
0927 04:59:27.368179 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:27.368186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:27.368288 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:27.368410 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:27.369770 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.371095 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.372352 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.373076 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.373223 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.377686 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:27.377707 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:27.377730 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:27.377781 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:27.381707 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:27.382346 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:27.382356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:27.382360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:27.382525 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:27.382886 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:27.382892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:27.382896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:27.383399 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:27.383710 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:27.383933 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 04:59:27.384172 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 04:59:27.385324 25678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:27.474849 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25555
I20260927 04:59:27.493376 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25688
Sep 27 04:59:27 dist-test-slave-39f2 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:27.512027 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:27.494178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.20.169.67:46123)
0927 04:59:27.494395 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:27.494401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:27.494425 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:59:27.494530 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:27.494539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:27.494650 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:27.494789 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:27.496041 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.497391 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.498823 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:27.499599 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:27.499732 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:27.504394 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:27.504408 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:27.504429 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:27.504491 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:27.508772 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:27.509583 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:27.509593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:27.509597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:27.509794 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:27.510203 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:27.510209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:27.510213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:27.510699 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:27.511048 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:27.511311 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 04:59:27.511552 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 04:59:27.512920 25811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:27.635541 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25688
I20260927 04:59:27.657294 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25328
I20260927 04:59:27.658517 25389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:27.767390 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25328
2026-09-27T04:59:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4410 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T04:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:27Z Disabled control of system clock
I20260927 04:59:27.836723 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33247
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 04:59:28.164184 25869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:28.164484 25869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:28.164626 25869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:28.172195 25869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:28.172307 25869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:28.172348 25869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:28.172448 25869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:28.172549 25869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:28.181821 25869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:33247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:28.183050 25869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:28.184891 25869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:28.196743 25875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:28.196861 25874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:28.196873 25877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:28.197866 25869 server_base.cc:1034] running on GCE node
I20260927 04:59:28.198696 25869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:28.200129 25869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:28.201359 25869 hybrid_clock.cc:648] HybridClock initialized: now 1790485168201300 us; error 57 us; skew 500 ppm
I20260927 04:59:28.201709 25869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:28.204013 25869 webserver.cc:460] Webserver started at http://127.20.169.126:37175/ using document root <none> and password file <none>
I20260927 04:59:28.204692 25869 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:28.204814 25869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:28.205219 25869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:28.208046 25869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "d9c4f5e19e2b4f7da481a064309b3214"
format_stamp: "Formatted at 2026-09-27 04:59:28 on dist-test-slave-39f2"
server_key: "c53673773d4612e73c6eaba6bf0627da"
server_key_iv: "1088efaf50fa3996d04df0ae223a1d93"
server_key_version: "encryptionkey@0"
I20260927 04:59:28.208799 25869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "d9c4f5e19e2b4f7da481a064309b3214"
format_stamp: "Formatted at 2026-09-27 04:59:28 on dist-test-slave-39f2"
server_key: "c53673773d4612e73c6eaba6bf0627da"
server_key_iv: "1088efaf50fa3996d04df0ae223a1d93"
server_key_version: "encryptionkey@0"
I20260927 04:59:28.213883 25869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 04:59:28.217245 25883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:28.218562 25869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 04:59:28.218745 25869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "d9c4f5e19e2b4f7da481a064309b3214"
format_stamp: "Formatted at 2026-09-27 04:59:28 on dist-test-slave-39f2"
server_key: "c53673773d4612e73c6eaba6bf0627da"
server_key_iv: "1088efaf50fa3996d04df0ae223a1d93"
server_key_version: "encryptionkey@0"
I20260927 04:59:28.218966 25869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:28.224684 25869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:28.225461 25869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:28.225754 25869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:28.244323 25869 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:33247
I20260927 04:59:28.244408 25934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:33247 every 8 connection(s)
I20260927 04:59:28.245698 25869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:28.249531 25935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:28.255395 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25869
I20260927 04:59:28.255617 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:28.256413 21157 external_mini_cluster.cc:1428] Setting key ef1c595d176c38cd1644818c952c0df0
I20260927 04:59:28.261159 25935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214: Bootstrap starting.
I20260927 04:59:28.264544 25935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:28.265055 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:28.258700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.20.169.126:33247)
0927 04:59:28.259675 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:28.259685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:28.259726 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:59:28.261436 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:28.261461 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:28.261869 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:28.262258 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:28.263345 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:28.263350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:28.263363 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:28.263377 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:28.263439 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:28.263454 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:28.263678 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:28.263815 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 04:59:28.264059 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":322,"threads_started":1}
I20260927 04:59:28.265728 25935 log.cc:824] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:28.268532 25935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214: No bootstrap required, opened a new log
I20260927 04:59:28.276042 25935 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c4f5e19e2b4f7da481a064309b3214" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33247 } }
I20260927 04:59:28.276417 25935 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:28.276535 25935 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c4f5e19e2b4f7da481a064309b3214, State: Initialized, Role: FOLLOWER
I20260927 04:59:28.277143 25935 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c4f5e19e2b4f7da481a064309b3214" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33247 } }
I20260927 04:59:28.277413 25935 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:28.277539 25935 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:28.277706 25935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:28.280187 25935 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c4f5e19e2b4f7da481a064309b3214" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33247 } }
I20260927 04:59:28.280690 25935 leader_election.cc:302] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9c4f5e19e2b4f7da481a064309b3214; no voters:
I20260927 04:59:28.281319 25935 leader_election.cc:288] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:28.281481 25940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:28.282161 25940 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [term 1 LEADER]: Becoming Leader. State: Replica: d9c4f5e19e2b4f7da481a064309b3214, State: Running, Role: LEADER
I20260927 04:59:28.282665 25940 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c4f5e19e2b4f7da481a064309b3214" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33247 } }
I20260927 04:59:28.283362 25935 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:28.285627 25942 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9c4f5e19e2b4f7da481a064309b3214. Latest consensus state: current_term: 1 leader_uuid: "d9c4f5e19e2b4f7da481a064309b3214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c4f5e19e2b4f7da481a064309b3214" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33247 } } }
I20260927 04:59:28.285835 25941 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9c4f5e19e2b4f7da481a064309b3214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c4f5e19e2b4f7da481a064309b3214" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33247 } } }
I20260927 04:59:28.286082 25942 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:28.286156 25941 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:28.289124 25949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:28.294175 25949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:28.302647 25949 catalog_manager.cc:1347] Generated new cluster ID: 4ae35dcc846649e39e96a0f620289bf3
I20260927 04:59:28.302827 25949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:28.331629 25949 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:28.332104 25949 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:28.342525 25949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214: Generated new TSK 0
I20260927 04:59:28.343510 25949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:59:28.350080 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33247
--builtin_ntp_servers=127.20.169.84:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 04:59:28.684100 25959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:28.684448 25959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:28.684620 25959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:28.692194 25959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:28.692345 25959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:28.692544 25959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:28.701915 25959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:28.703246 25959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:28.705068 25959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:28.718353 25967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:28.718434 25964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:28.718483 25965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:28.719189 25959 server_base.cc:1034] running on GCE node
I20260927 04:59:28.719775 25959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:28.720765 25959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:28.722015 25959 hybrid_clock.cc:648] HybridClock initialized: now 1790485168721957 us; error 66 us; skew 500 ppm
I20260927 04:59:28.722375 25959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:28.724787 25959 webserver.cc:460] Webserver started at http://127.20.169.65:40935/ using document root <none> and password file <none>
I20260927 04:59:28.725528 25959 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:28.725663 25959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:28.725996 25959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:28.728704 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "c96e8a836e924467a7a0278984eeeda9"
format_stamp: "Formatted at 2026-09-27 04:59:28 on dist-test-slave-39f2"
server_key: "75f043159f22f15003515642531c5bb6"
server_key_iv: "fdcc478f57b24d28e33db850c5042ddf"
server_key_version: "encryptionkey@0"
I20260927 04:59:28.729621 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "c96e8a836e924467a7a0278984eeeda9"
format_stamp: "Formatted at 2026-09-27 04:59:28 on dist-test-slave-39f2"
server_key: "75f043159f22f15003515642531c5bb6"
server_key_iv: "fdcc478f57b24d28e33db850c5042ddf"
server_key_version: "encryptionkey@0"
I20260927 04:59:28.734584 25959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 04:59:28.738358 25973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:28.739681 25959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 04:59:28.739856 25959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "c96e8a836e924467a7a0278984eeeda9"
format_stamp: "Formatted at 2026-09-27 04:59:28 on dist-test-slave-39f2"
server_key: "75f043159f22f15003515642531c5bb6"
server_key_iv: "fdcc478f57b24d28e33db850c5042ddf"
server_key_version: "encryptionkey@0"
I20260927 04:59:28.740082 25959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:28.745551 25959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:28.746277 25959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:28.746555 25959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:28.747437 25959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:28.748853 25959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:28.748993 25959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:28.749115 25959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:28.749200 25959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:28.782275 25959 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:43347
I20260927 04:59:28.782364 26086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:43347 every 8 connection(s)
I20260927 04:59:28.783717 25959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:28.791709 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 25959
I20260927 04:59:28.791965 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:28.792959 21157 external_mini_cluster.cc:1428] Setting key 5fda693fb508db7a297b7c687936719c
I20260927 04:59:28.792960 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:28.786545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:33007 (local address 127.20.169.126:33247)
0927 04:59:28.787038 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:28.787043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:28.788473 (+ 1430us) server_negotiation.cc:407] Connection header received
0927 04:59:28.790363 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:28.790367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:28.790427 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:28.790534 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:28.792021 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:28.792028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:28.792032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:28.792055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:28.792110 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:28.792131 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:28.792304 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:28.792473 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 04:59:28.792542 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260927 04:59:28.793846 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:28.787259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33247 (local address 127.20.169.65:33007)
0927 04:59:28.788311 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:28.788361 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:28.790051 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:28.790735 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:28.790743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:28.790833 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:28.791666 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:28.791671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:28.791726 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:28.791838 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:28.792260 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:28.792268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:28.792274 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:59:28.792494 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 04:59:28.792713 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":326,"threads_started":1}
I20260927 04:59:28.795931 26087 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33247
I20260927 04:59:28.796260 26087 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:28.796307 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33247
--builtin_ntp_servers=127.20.169.84:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 04:59:28.797127 26087 heartbeater.cc:511] Master 127.20.169.126:33247 requested a full tablet report, sending...
I20260927 04:59:28.799335 25900 ts_manager.cc:194] Registered new tserver with Master: c96e8a836e924467a7a0278984eeeda9 (127.20.169.65:43347)
W20260927 04:59:29.114732 26091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:29.115129 26091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:29.115283 26091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:29.122709 26091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:29.122815 26091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:29.122978 26091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:29.132508 26091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:29.133867 26091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:29.135689 26091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:29.149243 26097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:29.149194 26099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:29.149290 26096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:29.150261 26091 server_base.cc:1034] running on GCE node
I20260927 04:59:29.150895 26091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:29.151785 26091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:29.153008 26091 hybrid_clock.cc:648] HybridClock initialized: now 1790485169152969 us; error 50 us; skew 500 ppm
I20260927 04:59:29.153319 26091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:29.155541 26091 webserver.cc:460] Webserver started at http://127.20.169.66:36751/ using document root <none> and password file <none>
I20260927 04:59:29.156222 26091 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:29.156353 26091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:29.156687 26091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:29.159561 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "f4cba8f4ca1348f59907298212bea6b7"
format_stamp: "Formatted at 2026-09-27 04:59:29 on dist-test-slave-39f2"
server_key: "ee370f7e2a66c90417ba08d4955d3aab"
server_key_iv: "2fcccc5f1bef71e796d8648ad29220ea"
server_key_version: "encryptionkey@0"
I20260927 04:59:29.160367 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "f4cba8f4ca1348f59907298212bea6b7"
format_stamp: "Formatted at 2026-09-27 04:59:29 on dist-test-slave-39f2"
server_key: "ee370f7e2a66c90417ba08d4955d3aab"
server_key_iv: "2fcccc5f1bef71e796d8648ad29220ea"
server_key_version: "encryptionkey@0"
I20260927 04:59:29.165758 26091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 04:59:29.169111 26105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:29.170383 26091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 04:59:29.170563 26091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "f4cba8f4ca1348f59907298212bea6b7"
format_stamp: "Formatted at 2026-09-27 04:59:29 on dist-test-slave-39f2"
server_key: "ee370f7e2a66c90417ba08d4955d3aab"
server_key_iv: "2fcccc5f1bef71e796d8648ad29220ea"
server_key_version: "encryptionkey@0"
I20260927 04:59:29.170778 26091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:29.176476 26091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:29.177294 26091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:29.177564 26091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:29.178439 26091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:29.179829 26091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:29.179949 26091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:29.180074 26091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:29.180158 26091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:29.213579 26091 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:39935
I20260927 04:59:29.213620 26218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:39935 every 8 connection(s)
I20260927 04:59:29.214928 26091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:29.217163 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26091
I20260927 04:59:29.217396 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:29.217898 21157 external_mini_cluster.cc:1428] Setting key c41d2554004ce32e3d9022febf771081
I20260927 04:59:29.220645 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33247
--builtin_ntp_servers=127.20.169.84:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 04:59:29.223788 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.217633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:56315 (local address 127.20.169.126:33247)
0927 04:59:29.217852 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.217858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.219473 (+ 1615us) server_negotiation.cc:407] Connection header received
0927 04:59:29.221271 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.221277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.221343 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.221462 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.222985 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.222991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.222995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.223015 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.223053 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.223064 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.223201 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.223391 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.223459 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 04:59:29.224680 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.218254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33247 (local address 127.20.169.66:56315)
0927 04:59:29.219305 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.219349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.221012 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.221711 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.221723 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.221831 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.222573 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.222577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.222653 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.222750 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.223158 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.223171 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.223180 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.223481 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.223739 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":334,"threads_started":1}
I20260927 04:59:29.226068 26219 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33247
I20260927 04:59:29.226404 26219 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:29.227126 26219 heartbeater.cc:511] Master 127.20.169.126:33247 requested a full tablet report, sending...
I20260927 04:59:29.228471 25900 ts_manager.cc:194] Registered new tserver with Master: f4cba8f4ca1348f59907298212bea6b7 (127.20.169.66:39935)
W20260927 04:59:29.549343 26222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:29.549664 26222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:29.549849 26222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:29.557286 26222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:29.557415 26222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:29.557595 26222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:29.566741 26222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:29.568075 26222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:29.569988 26222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:29.583158 26227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:29.583199 26230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:29.583223 26228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:29.584517 26222 server_base.cc:1034] running on GCE node
I20260927 04:59:29.585201 26222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:29.586200 26222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:29.587441 26222 hybrid_clock.cc:648] HybridClock initialized: now 1790485169587386 us; error 39 us; skew 500 ppm
I20260927 04:59:29.587811 26222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:29.590257 26222 webserver.cc:460] Webserver started at http://127.20.169.67:39521/ using document root <none> and password file <none>
I20260927 04:59:29.590946 26222 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:29.591073 26222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:29.591435 26222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:29.594460 26222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "b5612e04c80f41a59679b30ba4411047"
format_stamp: "Formatted at 2026-09-27 04:59:29 on dist-test-slave-39f2"
server_key: "4e3a0ebfbc7e3cf3090b18c6f905da9c"
server_key_iv: "a49b1573938dc4b34dadcf62a19dc88a"
server_key_version: "encryptionkey@0"
I20260927 04:59:29.595341 26222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "b5612e04c80f41a59679b30ba4411047"
format_stamp: "Formatted at 2026-09-27 04:59:29 on dist-test-slave-39f2"
server_key: "4e3a0ebfbc7e3cf3090b18c6f905da9c"
server_key_iv: "a49b1573938dc4b34dadcf62a19dc88a"
server_key_version: "encryptionkey@0"
I20260927 04:59:29.600605 26222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 04:59:29.604138 26236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:29.605654 26222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:29.605824 26222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "b5612e04c80f41a59679b30ba4411047"
format_stamp: "Formatted at 2026-09-27 04:59:29 on dist-test-slave-39f2"
server_key: "4e3a0ebfbc7e3cf3090b18c6f905da9c"
server_key_iv: "a49b1573938dc4b34dadcf62a19dc88a"
server_key_version: "encryptionkey@0"
I20260927 04:59:29.606009 26222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:29.611678 26222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:29.612385 26222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:29.612622 26222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:29.613576 26222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:29.614925 26222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:29.615008 26222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:29.615087 26222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:29.615151 26222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:29.648207 26222 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:38109
I20260927 04:59:29.648258 26348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:38109 every 8 connection(s)
I20260927 04:59:29.649497 26222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:29.650710 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26222
I20260927 04:59:29.650892 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:29.651413 21157 external_mini_cluster.cc:1428] Setting key 64102495965416d9232132ecd32ff0b6
I20260927 04:59:29.658685 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.652446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:53753 (local address 127.20.169.126:33247)
0927 04:59:29.652800 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.652805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.654578 (+ 1773us) server_negotiation.cc:407] Connection header received
0927 04:59:29.656353 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.656357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.656411 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.656507 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.657989 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.657993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.657996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.658011 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.658043 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.658054 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.658154 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.658383 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.658445 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260927 04:59:29.659570 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.653188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33247 (local address 127.20.169.67:53753)
0927 04:59:29.654423 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.654471 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.656081 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.656716 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.656724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.656891 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.657666 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.657668 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.657708 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.657827 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.658122 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.658128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.658132 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.658344 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.658553 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":369,"threads_started":1}
I20260927 04:59:29.661237 26349 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33247
I20260927 04:59:29.661579 26349 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:29.662161 26349 heartbeater.cc:511] Master 127.20.169.126:33247 requested a full tablet report, sending...
I20260927 04:59:29.663574 25900 ts_manager.cc:194] Registered new tserver with Master: b5612e04c80f41a59679b30ba4411047 (127.20.169.67:38109)
I20260927 04:59:29.666868 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:59:29.674016 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.671395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.20.169.126:33247)
0927 04:59:29.671588 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.671592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.672143 (+ 551us) server_negotiation.cc:407] Connection header received
0927 04:59:29.672274 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.672278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.672374 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.672461 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.673405 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.673410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.673413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.673429 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.673462 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.673474 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.673576 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.673669 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.673726 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 04:59:29.678190 25900 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:29.685253 25900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:29.706054 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.698265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:38109 (local address 127.0.0.1:59890)
0927 04:59:29.700487 (+ 2222us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.700526 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.700673 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.703037 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.703042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.703073 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.704229 (+ 1156us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.704233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.704268 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.704360 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.705058 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.705080 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.705091 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.705326 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.705436 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1972,"thread_start_us":1802,"threads_started":1}
I20260927 04:59:29.706507 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.701458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.20.169.66:39935)
0927 04:59:29.702739 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.702750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.703832 (+ 1082us) server_negotiation.cc:407] Connection header received
0927 04:59:29.704109 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.704120 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.704320 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.704516 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.705672 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.705678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.705686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.705704 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.705739 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.705752 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.705954 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.706033 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.706138 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":952,"threads_started":1}
I20260927 04:59:29.706900 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.698593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.20.169.67:38109)
0927 04:59:29.699896 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.699909 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.700880 (+ 971us) server_negotiation.cc:407] Connection header received
0927 04:59:29.700965 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.700973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.701644 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.701844 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.704613 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.704622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.704633 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.704658 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.704729 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.704754 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.704977 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.706338 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.706448 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":903,"threads_started":1}
I20260927 04:59:29.707089 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.699557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.20.169.65:43347)
0927 04:59:29.700482 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.700496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.701375 (+ 879us) server_negotiation.cc:407] Connection header received
0927 04:59:29.701469 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.701478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.701719 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.701889 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.704605 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.704616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.704633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.704664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.704729 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.704754 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.704938 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.706544 (+ 1606us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.706639 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":1618,"threads_started":1}
I20260927 04:59:29.706915 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.700962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:39935 (local address 127.0.0.1:41334)
0927 04:59:29.703691 (+ 2729us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.703710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.703861 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.704509 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.704514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.704539 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.705327 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.705330 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.705352 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.705398 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.705875 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.705881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.705886 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.706060 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.706114 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2554,"thread_start_us":2366,"threads_started":1}
I20260927 04:59:29.708081 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.699137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40416)
0927 04:59:29.701065 (+ 1928us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.701089 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.701215 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.702962 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.702975 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.703074 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.704252 (+ 1178us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.704256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.704269 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.704360 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.706335 (+ 1975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.706341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.706347 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.707431 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.707517 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":1545,"threads_started":1}
I20260927 04:59:29.711621 26022 tablet_service.cc:1467] Processing CreateTablet for tablet e7494de0b3c748ca99edc5f2a5cde495 (DEFAULT_TABLE table=test-table [id=628d4bbeb8974debbf22ee08e14141ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:29.711621 26154 tablet_service.cc:1467] Processing CreateTablet for tablet e7494de0b3c748ca99edc5f2a5cde495 (DEFAULT_TABLE table=test-table [id=628d4bbeb8974debbf22ee08e14141ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:29.713238 26022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7494de0b3c748ca99edc5f2a5cde495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:29.713238 26154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7494de0b3c748ca99edc5f2a5cde495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:29.714834 26284 tablet_service.cc:1467] Processing CreateTablet for tablet e7494de0b3c748ca99edc5f2a5cde495 (DEFAULT_TABLE table=test-table [id=628d4bbeb8974debbf22ee08e14141ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:29.715999 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7494de0b3c748ca99edc5f2a5cde495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:29.727630 26367 tablet_bootstrap.cc:492] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Bootstrap starting.
I20260927 04:59:29.729589 26369 tablet_bootstrap.cc:492] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Bootstrap starting.
I20260927 04:59:29.731690 26368 tablet_bootstrap.cc:492] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Bootstrap starting.
I20260927 04:59:29.731964 26367 tablet_bootstrap.cc:654] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:29.733289 26367 log.cc:824] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:29.733897 26369 tablet_bootstrap.cc:654] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:29.734962 26368 tablet_bootstrap.cc:654] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:29.735142 26369 log.cc:824] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:29.736084 26368 log.cc:824] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:29.736999 26367 tablet_bootstrap.cc:492] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: No bootstrap required, opened a new log
I20260927 04:59:29.737323 26367 ts_tablet_manager.cc:1397] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 04:59:29.738523 26369 tablet_bootstrap.cc:492] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: No bootstrap required, opened a new log
I20260927 04:59:29.738762 26368 tablet_bootstrap.cc:492] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: No bootstrap required, opened a new log
I20260927 04:59:29.738870 26369 ts_tablet_manager.cc:1397] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260927 04:59:29.739058 26368 ts_tablet_manager.cc:1397] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260927 04:59:29.747206 26368 raft_consensus.cc:348] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.747605 26368 raft_consensus.cc:374] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:29.747771 26368 raft_consensus.cc:729] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cba8f4ca1348f59907298212bea6b7, State: Initialized, Role: FOLLOWER
I20260927 04:59:29.748427 26368 consensus_queue.cc:260] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.749708 26367 raft_consensus.cc:348] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.750309 26367 raft_consensus.cc:374] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:29.750473 26367 raft_consensus.cc:729] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5612e04c80f41a59679b30ba4411047, State: Initialized, Role: FOLLOWER
I20260927 04:59:29.751437 26219 heartbeater.cc:503] Master 127.20.169.126:33247 was elected leader, sending a full tablet report...
I20260927 04:59:29.751157 26369 raft_consensus.cc:348] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.751245 26367 consensus_queue.cc:260] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.751621 26369 raft_consensus.cc:374] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:29.751784 26369 raft_consensus.cc:729] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96e8a836e924467a7a0278984eeeda9, State: Initialized, Role: FOLLOWER
I20260927 04:59:29.752194 26368 ts_tablet_manager.cc:1428] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260927 04:59:29.752678 26369 consensus_queue.cc:260] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.753523 26349 heartbeater.cc:503] Master 127.20.169.126:33247 was elected leader, sending a full tablet report...
I20260927 04:59:29.754593 26367 ts_tablet_manager.cc:1428] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260927 04:59:29.755420 26087 heartbeater.cc:503] Master 127.20.169.126:33247 was elected leader, sending a full tablet report...
I20260927 04:59:29.756253 26369 ts_tablet_manager.cc:1428] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
W20260927 04:59:29.786010 26088 tablet.cc:2369] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:59:29.901157 26350 tablet.cc:2369] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:29.941269 26375 raft_consensus.cc:482] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:29.941587 26375 raft_consensus.cc:504] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.943296 26375 leader_election.cc:288] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5612e04c80f41a59679b30ba4411047 (127.20.169.67:38109), f4cba8f4ca1348f59907298212bea6b7 (127.20.169.66:39935)
I20260927 04:59:29.947063 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.943691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:56609 (local address 127.20.169.67:38109)
0927 04:59:29.943865 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.943869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.944688 (+ 819us) server_negotiation.cc:407] Connection header received
0927 04:59:29.945069 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.945074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.945143 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.945265 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.946400 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.946406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.946410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.946430 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.946466 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.946484 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.946603 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.946697 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.946767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 04:59:29.947281 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.944698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:43281 (local address 127.20.169.66:39935)
0927 04:59:29.944903 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:29.944909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:29.945328 (+ 419us) server_negotiation.cc:407] Connection header received
0927 04:59:29.945466 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:29.945472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:29.945536 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:29.945647 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:29.946611 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:29.946616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:29.946620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:29.946641 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:29.946681 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:29.946698 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:29.946824 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:29.946959 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 04:59:29.947018 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:59:29.947504 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.943534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:38109 (local address 127.20.169.65:56609)
0927 04:59:29.944223 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.944243 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.944879 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.945324 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.945329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.945358 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.946163 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.946165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.946179 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.946217 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.946587 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.946592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.946597 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.946741 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.946798 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":330,"threads_started":1}
I20260927 04:59:29.947698 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:29.944332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:39935 (local address 127.20.169.65:43281)
0927 04:59:29.945216 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:29.945237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:29.945340 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:29.945669 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:29.945672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:29.945688 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:29.946388 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:59:29.946391 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:29.946408 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:59:29.946450 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:29.946803 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:29.946807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:29.946812 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:59:29.946984 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 04:59:29.947031 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":572,"threads_started":1}
I20260927 04:59:29.948596 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7494de0b3c748ca99edc5f2a5cde495" candidate_uuid: "c96e8a836e924467a7a0278984eeeda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5612e04c80f41a59679b30ba4411047" is_pre_election: true
I20260927 04:59:29.948761 26174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7494de0b3c748ca99edc5f2a5cde495" candidate_uuid: "c96e8a836e924467a7a0278984eeeda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cba8f4ca1348f59907298212bea6b7" is_pre_election: true
I20260927 04:59:29.949143 26304 raft_consensus.cc:2454] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c96e8a836e924467a7a0278984eeeda9 in term 0.
I20260927 04:59:29.949307 26174 raft_consensus.cc:2454] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c96e8a836e924467a7a0278984eeeda9 in term 0.
I20260927 04:59:29.949928 25977 leader_election.cc:302] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5612e04c80f41a59679b30ba4411047, c96e8a836e924467a7a0278984eeeda9; no voters:
I20260927 04:59:29.950385 26375 raft_consensus.cc:2781] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:29.950531 26375 raft_consensus.cc:482] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:29.950636 26375 raft_consensus.cc:3037] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:29.952744 26375 raft_consensus.cc:504] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.953413 26375 leader_election.cc:288] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [CANDIDATE]: Term 1 election: Requested vote from peers b5612e04c80f41a59679b30ba4411047 (127.20.169.67:38109), f4cba8f4ca1348f59907298212bea6b7 (127.20.169.66:39935)
I20260927 04:59:29.953794 26304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7494de0b3c748ca99edc5f2a5cde495" candidate_uuid: "c96e8a836e924467a7a0278984eeeda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5612e04c80f41a59679b30ba4411047"
I20260927 04:59:29.953950 26174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7494de0b3c748ca99edc5f2a5cde495" candidate_uuid: "c96e8a836e924467a7a0278984eeeda9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cba8f4ca1348f59907298212bea6b7"
I20260927 04:59:29.953985 26304 raft_consensus.cc:3037] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:29.954156 26174 raft_consensus.cc:3037] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:29.956132 26304 raft_consensus.cc:2454] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c96e8a836e924467a7a0278984eeeda9 in term 1.
I20260927 04:59:29.956367 26174 raft_consensus.cc:2454] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c96e8a836e924467a7a0278984eeeda9 in term 1.
I20260927 04:59:29.956526 25977 leader_election.cc:302] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5612e04c80f41a59679b30ba4411047, c96e8a836e924467a7a0278984eeeda9; no voters:
I20260927 04:59:29.956790 26375 raft_consensus.cc:2781] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:29.957262 26375 raft_consensus.cc:686] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 1 LEADER]: Becoming Leader. State: Replica: c96e8a836e924467a7a0278984eeeda9, State: Running, Role: LEADER
I20260927 04:59:29.957728 26375 consensus_queue.cc:237] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } }
I20260927 04:59:29.962314 25900 catalog_manager.cc:5495] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c96e8a836e924467a7a0278984eeeda9 (127.20.169.65). New cstate: current_term: 1 leader_uuid: "c96e8a836e924467a7a0278984eeeda9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c96e8a836e924467a7a0278984eeeda9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43347 } health_report { overall_health: HEALTHY } } }
W20260927 04:59:29.966918 26220 tablet.cc:2369] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:30.004037 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.001539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.20.169.65:43347)
0927 04:59:30.001743 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:30.001749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:30.001951 (+ 202us) server_negotiation.cc:407] Connection header received
0927 04:59:30.002214 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:30.002218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:30.002301 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:30.002416 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:30.003353 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:30.003359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:30.003364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:30.003390 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:30.003422 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:30.003435 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:30.003561 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:30.003687 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 04:59:30.003752 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 04:59:30.012434 26304 raft_consensus.cc:1261] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 1 FOLLOWER]: Refusing update from remote peer c96e8a836e924467a7a0278984eeeda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:30.012653 26174 raft_consensus.cc:1261] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 1 FOLLOWER]: Refusing update from remote peer c96e8a836e924467a7a0278984eeeda9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:30.013521 26375 consensus_queue.cc:1035] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5612e04c80f41a59679b30ba4411047" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:30.013885 26378 consensus_queue.cc:1035] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4cba8f4ca1348f59907298212bea6b7" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 39935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:30.038928 25900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:30.039216 25900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:30.042770 25900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9c4f5e19e2b4f7da481a064309b3214: Sending DeleteTablet for 3 replicas of tablet e7494de0b3c748ca99edc5f2a5cde495
I20260927 04:59:30.043962 26022 tablet_service.cc:1514] Processing DeleteTablet for tablet e7494de0b3c748ca99edc5f2a5cde495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:30 UTC) from {username='slave'} at 127.0.0.1:40416
I20260927 04:59:30.044018 26284 tablet_service.cc:1514] Processing DeleteTablet for tablet e7494de0b3c748ca99edc5f2a5cde495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:30 UTC) from {username='slave'} at 127.0.0.1:59890
I20260927 04:59:30.044119 26154 tablet_service.cc:1514] Processing DeleteTablet for tablet e7494de0b3c748ca99edc5f2a5cde495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:30 UTC) from {username='slave'} at 127.0.0.1:41334
I20260927 04:59:30.046129 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25959
I20260927 04:59:30.057300 26393 tablet_replica.cc:330] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: stopping tablet replica
I20260927 04:59:30.057688 26393 raft_consensus.cc:2229] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:30.058132 26393 raft_consensus.cc:2258] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:30.062619 26393 ts_tablet_manager.cc:1905] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:30.069643 26393 ts_tablet_manager.cc:1918] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:30.069876 26393 log.cc:1196] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/e7494de0b3c748ca99edc5f2a5cde495
I20260927 04:59:30.070533 26393 ts_tablet_manager.cc:1939] T e7494de0b3c748ca99edc5f2a5cde495 P b5612e04c80f41a59679b30ba4411047: Deleting consensus metadata
I20260927 04:59:30.072501 25887 catalog_manager.cc:4841] TS b5612e04c80f41a59679b30ba4411047 (127.20.169.67:38109): tablet e7494de0b3c748ca99edc5f2a5cde495 (table test-table [id=628d4bbeb8974debbf22ee08e14141ed]) successfully deleted
I20260927 04:59:30.073596 26391 tablet_replica.cc:330] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: stopping tablet replica
I20260927 04:59:30.074143 26391 raft_consensus.cc:2229] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:30.074937 26391 raft_consensus.cc:2258] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:30.076306 26392 tablet_replica.cc:330] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: stopping tablet replica
I20260927 04:59:30.076778 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.047846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.20.169.65:43347)
0927 04:59:30.048077 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:30.048086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:30.071571 (+ 23485us) server_negotiation.cc:407] Connection header received
0927 04:59:30.072175 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:30.072185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:30.072275 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:30.072370 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:30.075777 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:30.075782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:30.075787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:30.075809 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:30.075849 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:30.075866 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:30.076000 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:30.076227 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 04:59:30.076289 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 04:59:30.076756 26392 raft_consensus.cc:2229] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:30.077900 26082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:30.078326 26392 raft_consensus.cc:2258] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:30.080577 26392 ts_tablet_manager.cc:1905] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:30.085803 26392 ts_tablet_manager.cc:1918] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:30.085949 26392 log.cc:1196] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/e7494de0b3c748ca99edc5f2a5cde495
I20260927 04:59:30.086396 26392 ts_tablet_manager.cc:1939] T e7494de0b3c748ca99edc5f2a5cde495 P f4cba8f4ca1348f59907298212bea6b7: Deleting consensus metadata
I20260927 04:59:30.087929 25887 catalog_manager.cc:4841] TS f4cba8f4ca1348f59907298212bea6b7 (127.20.169.66:39935): tablet e7494de0b3c748ca99edc5f2a5cde495 (table test-table [id=628d4bbeb8974debbf22ee08e14141ed]) successfully deleted
I20260927 04:59:30.078085 26391 ts_tablet_manager.cc:1905] T e7494de0b3c748ca99edc5f2a5cde495 P c96e8a836e924467a7a0278984eeeda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:30.172788 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25959
I20260927 04:59:30.194715 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26091
W20260927 04:59:30.194890 25885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:43347 (error 108)
I20260927 04:59:30.196609 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.196096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40428)
0927 04:59:30.196284 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.196438 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 04:59:30.196972 25885 catalog_manager.cc:4584] TS c96e8a836e924467a7a0278984eeeda9 (127.20.169.65:43347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7494de0b3c748ca99edc5f2a5cde495: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
I20260927 04:59:30.197682 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.195585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.20.169.66:39935)
0927 04:59:30.195765 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:30.195771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:30.195793 (+ 22us) server_negotiation.cc:407] Connection header received
0927 04:59:30.195901 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:30.195908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:30.196007 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:30.196093 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:30.197115 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:30.197123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:30.197128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:30.197146 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:30.197188 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:30.197204 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:30.197338 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:30.197443 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 04:59:30.197488 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 04:59:30.198303 26214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:30.250828 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.250301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40434)
0927 04:59:30.250540 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.250649 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 04:59:30.252010 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.251604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40444)
0927 04:59:30.251767 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.251871 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 04:59:30.288111 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26091
I20260927 04:59:30.305662 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26222
I20260927 04:59:30.308509 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.306547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.20.169.67:38109)
0927 04:59:30.306712 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:30.306716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:30.306735 (+ 19us) server_negotiation.cc:407] Connection header received
0927 04:59:30.306856 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:30.306860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:30.306951 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:30.307040 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:30.307950 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:30.307955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:30.307958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:30.307975 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:30.308009 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:30.308022 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:30.308133 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:30.308198 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 04:59:30.308248 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 04:59:30.309242 26344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:30.312402 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.311891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40456)
0927 04:59:30.312078 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.312196 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 04:59:30.313596 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.313184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40460)
0927 04:59:30.313337 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.313438 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 04:59:30.394336 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.393747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40466)
0927 04:59:30.394012 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.394133 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 04:59:30.395534 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:30.395074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43347 (local address 127.0.0.1:40480)
0927 04:59:30.395290 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:30.395373 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 04:59:30.397733 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26222
I20260927 04:59:30.415043 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 25869
I20260927 04:59:30.416386 25930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:30.522185 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 25869
2026-09-27T04:59:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2753 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 04:59:30.564429 21157 mini_oidc.cc:152] Starting JWKS server
I20260927 04:59:30.567528 21157 webserver.cc:460] Webserver started at https://127.0.0.1:45679/ using document root <none> and password file <none>
I20260927 04:59:30.567893 21157 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 04:59:30.568718 21157 webserver.cc:460] Webserver started at http://127.0.0.1:35275/ using document root <none> and password file <none>
2026-09-27T04:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:30Z Disabled control of system clock
I20260927 04:59:30.598104 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33261
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:44067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:33261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/testchainca.cert with env {}
W20260927 04:59:30.915932 26406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:30.916209 26406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:30.916296 26406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:30.923410 26406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:30.923518 26406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:30.923556 26406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:30.923604 26406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:30.923651 26406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:59:30.923712 26406 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:59:30.923750 26406 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45679/jwks.json
I20260927 04:59:30.933642 26406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:33261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:30.934940 26406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:30.936162 26406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:30.948386 26411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:30.948782 26412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:30.949288 26414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:30.950119 26406 server_base.cc:1034] running on GCE node
I20260927 04:59:30.950846 26406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:30.952214 26406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:30.953410 26406 hybrid_clock.cc:648] HybridClock initialized: now 1790485170953375 us; error 24 us; skew 500 ppm
I20260927 04:59:30.953775 26406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:30.956121 26406 webserver.cc:460] Webserver started at http://127.20.169.126:45121/ using document root <none> and password file <none>
I20260927 04:59:30.956799 26406 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:30.956965 26406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:30.957302 26406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:30.960172 26406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "5f6e81d408fc4e05b66bdc1f02551b2a"
format_stamp: "Formatted at 2026-09-27 04:59:30 on dist-test-slave-39f2"
server_key: "5b542d1aa69374502933e7e2733ed21b"
server_key_iv: "ddaaa36afcca88a18e570baa82c0c197"
server_key_version: "encryptionkey@0"
I20260927 04:59:30.961056 26406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "5f6e81d408fc4e05b66bdc1f02551b2a"
format_stamp: "Formatted at 2026-09-27 04:59:30 on dist-test-slave-39f2"
server_key: "5b542d1aa69374502933e7e2733ed21b"
server_key_iv: "ddaaa36afcca88a18e570baa82c0c197"
server_key_version: "encryptionkey@0"
I20260927 04:59:30.965965 26406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 04:59:30.969547 26420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:30.970922 26406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 04:59:30.971127 26406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "5f6e81d408fc4e05b66bdc1f02551b2a"
format_stamp: "Formatted at 2026-09-27 04:59:30 on dist-test-slave-39f2"
server_key: "5b542d1aa69374502933e7e2733ed21b"
server_key_iv: "ddaaa36afcca88a18e570baa82c0c197"
server_key_version: "encryptionkey@0"
I20260927 04:59:30.971349 26406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:30.987058 26406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:30.987856 26406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:30.988173 26406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:31.009665 26406 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:33261
I20260927 04:59:31.009686 26472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:33261 every 8 connection(s)
I20260927 04:59:31.011128 26406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:31.015152 26473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:31.016566 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26406
I20260927 04:59:31.016769 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:31.017460 21157 external_mini_cluster.cc:1428] Setting key 717e07308cb95e7a0319cdc85914f831
I20260927 04:59:31.028177 26473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a: Bootstrap starting.
I20260927 04:59:31.030501 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:31.019407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.20.169.126:33261)
0927 04:59:31.020410 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:31.020421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:31.020463 (+ 42us) server_negotiation.cc:407] Connection header received
0927 04:59:31.022079 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:31.022102 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:31.022597 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:31.022985 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:31.023528 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.024728 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.026071 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.026720 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.026929 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:31.028149 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:31.028156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:31.028174 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:31.028742 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:31.028877 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:31.028905 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:31.029042 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:31.029265 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 04:59:31.029571 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":346,"threads_started":1}
I20260927 04:59:31.031883 26473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:31.033084 26473 log.cc:824] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:31.034547 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:59:31.036388 26473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a: No bootstrap required, opened a new log
I20260927 04:59:31.044039 26473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33261 } }
I20260927 04:59:31.044452 26473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:31.044529 26473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6e81d408fc4e05b66bdc1f02551b2a, State: Initialized, Role: FOLLOWER
I20260927 04:59:31.045112 26473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33261 } }
I20260927 04:59:31.045310 26473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:31.045409 26473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:31.045533 26473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:31.047609 26473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33261 } }
I20260927 04:59:31.048069 26473 leader_election.cc:302] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f6e81d408fc4e05b66bdc1f02551b2a; no voters:
I20260927 04:59:31.048594 26473 leader_election.cc:288] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:31.048763 26478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:31.049577 26478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [term 1 LEADER]: Becoming Leader. State: Replica: 5f6e81d408fc4e05b66bdc1f02551b2a, State: Running, Role: LEADER
I20260927 04:59:31.050096 26478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33261 } }
I20260927 04:59:31.050909 26473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:31.053057 26479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33261 } } }
I20260927 04:59:31.053402 26479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:31.054245 26484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:31.054126 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f6e81d408fc4e05b66bdc1f02551b2a. Latest consensus state: current_term: 1 leader_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f6e81d408fc4e05b66bdc1f02551b2a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33261 } } }
I20260927 04:59:31.055379 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:31.060958 26484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:31.070363 26484 catalog_manager.cc:1347] Generated new cluster ID: 224a96319fd544a3b483ef0b5880f9f5
I20260927 04:59:31.070539 26484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:31.096107 26484 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:31.097620 26484 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:31.106483 26484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f6e81d408fc4e05b66bdc1f02551b2a: Generated new TSK 0
I20260927 04:59:31.107621 26484 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 04:59:31.153139 26503 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 04:59:31.153402 26503 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 04:59:31.154395 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:31.129873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.20.169.126:33261)
0927 04:59:31.130119 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:31.130131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:31.130753 (+ 622us) server_negotiation.cc:407] Connection header received
0927 04:59:31.131124 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:31.131129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:31.131279 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:31.131506 (+ 227us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:59:31.132134 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.133062 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.135169 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.135717 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.135844 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:31.136514 (+ 670us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:59:31.153830 (+ 17316us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 04:59:31.154142 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33568: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 04:59:31.154556 26476 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33568: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 04:59:31.156682 26502 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:31.129886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33261 (local address 127.0.0.1:33568)
0927 04:59:31.130622 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:31.130642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:31.130813 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:31.131534 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:31.131540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:31.131554 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:59:31.131942 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:31.131954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.133196 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.133202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:31.135000 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:31.135019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.135916 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.135923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:31.136109 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:31.136181 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:59:31.154145 (+ 17964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 04:59:31.154527 (+ 382us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.169.126:33261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":376,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.169.126:33261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 04:59:31.158775 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26406
I20260927 04:59:31.160115 26468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:31.254355 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26406
2026-09-27T04:59:31Z chronyd exiting
I20260927 04:59:31.306227 21157 test_util.cc:175] -----------------------------------------------
I20260927 04:59:31.306344 21157 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485126081201-21157-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (746 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 04:59:31.310098 21157 mini_oidc.cc:152] Starting JWKS server
I20260927 04:59:31.311908 21157 webserver.cc:460] Webserver started at https://127.0.0.1:42027/ using document root <none> and password file <none>
I20260927 04:59:31.312176 21157 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 04:59:31.312799 21157 webserver.cc:460] Webserver started at http://127.0.0.1:41131/ using document root <none> and password file <none>
2026-09-27T04:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:31Z Disabled control of system clock
I20260927 04:59:31.339620 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35319
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35319
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/testchainca.cert with env {}
W20260927 04:59:31.672727 26510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:31.673097 26510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:31.673271 26510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:31.680603 26510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:31.680737 26510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:31.680848 26510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:31.680962 26510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:31.681046 26510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:59:31.681135 26510 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:59:31.681213 26510 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42027/jwks.json
I20260927 04:59:31.691054 26510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:31.692358 26510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:31.693692 26510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:31.705221 26519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:31.705221 26517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:31.705806 26516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:31.706517 26510 server_base.cc:1034] running on GCE node
I20260927 04:59:31.707397 26510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:31.708775 26510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:31.710050 26510 hybrid_clock.cc:648] HybridClock initialized: now 1790485171709996 us; error 36 us; skew 500 ppm
I20260927 04:59:31.710407 26510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:31.718459 26510 webserver.cc:460] Webserver started at http://127.20.169.126:46815/ using document root <none> and password file <none>
I20260927 04:59:31.719169 26510 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:31.719313 26510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:31.719645 26510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:31.722606 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "4978301f9fbe4a53b8f556f31db457eb"
format_stamp: "Formatted at 2026-09-27 04:59:31 on dist-test-slave-39f2"
server_key: "a54ef7ef3f4b0420bd8578bd7d89466e"
server_key_iv: "578391adcfe248eb971ebfd70633580e"
server_key_version: "encryptionkey@0"
I20260927 04:59:31.723407 26510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "4978301f9fbe4a53b8f556f31db457eb"
format_stamp: "Formatted at 2026-09-27 04:59:31 on dist-test-slave-39f2"
server_key: "a54ef7ef3f4b0420bd8578bd7d89466e"
server_key_iv: "578391adcfe248eb971ebfd70633580e"
server_key_version: "encryptionkey@0"
I20260927 04:59:31.728751 26510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 04:59:31.732134 26525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:31.733568 26510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:31.733757 26510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "4978301f9fbe4a53b8f556f31db457eb"
format_stamp: "Formatted at 2026-09-27 04:59:31 on dist-test-slave-39f2"
server_key: "a54ef7ef3f4b0420bd8578bd7d89466e"
server_key_iv: "578391adcfe248eb971ebfd70633580e"
server_key_version: "encryptionkey@0"
I20260927 04:59:31.733974 26510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:31.755578 26510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:31.756520 26510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:31.756922 26510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:31.775357 26510 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35319
I20260927 04:59:31.775363 26576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35319 every 8 connection(s)
I20260927 04:59:31.776722 26510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:31.777164 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26510
I20260927 04:59:31.777400 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:31.778275 21157 external_mini_cluster.cc:1428] Setting key 8f64ddc515612e0a97af529757a36c44
I20260927 04:59:31.781519 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:31.791496 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:31.780179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.20.169.126:35319)
0927 04:59:31.781248 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:31.781259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:31.781301 (+ 42us) server_negotiation.cc:407] Connection header received
0927 04:59:31.782929 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:31.782950 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:31.783390 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:31.783806 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:31.784422 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.785797 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.787131 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.787803 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.788016 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:31.789304 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:31.789312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:31.789331 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:31.789878 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:31.789955 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:31.789978 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:31.790143 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:31.790347 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 04:59:31.790619 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":374,"threads_started":1}
I20260927 04:59:31.792100 26577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb: Bootstrap starting.
I20260927 04:59:31.795186 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:59:31.795444 26577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:31.796417 26577 log.cc:824] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:31.799598 26577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb: No bootstrap required, opened a new log
I20260927 04:59:31.807188 26577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4978301f9fbe4a53b8f556f31db457eb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35319 } }
I20260927 04:59:31.807543 26577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:31.807623 26577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4978301f9fbe4a53b8f556f31db457eb, State: Initialized, Role: FOLLOWER
I20260927 04:59:31.808130 26577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4978301f9fbe4a53b8f556f31db457eb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35319 } }
I20260927 04:59:31.808322 26577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:31.808422 26577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:31.808553 26577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:31.810751 26577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4978301f9fbe4a53b8f556f31db457eb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35319 } }
I20260927 04:59:31.811290 26577 leader_election.cc:302] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4978301f9fbe4a53b8f556f31db457eb; no voters:
I20260927 04:59:31.811857 26577 leader_election.cc:288] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:31.811998 26582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:31.812744 26582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [term 1 LEADER]: Becoming Leader. State: Replica: 4978301f9fbe4a53b8f556f31db457eb, State: Running, Role: LEADER
I20260927 04:59:31.813288 26582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4978301f9fbe4a53b8f556f31db457eb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35319 } }
I20260927 04:59:31.813988 26577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:31.816326 26584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4978301f9fbe4a53b8f556f31db457eb. Latest consensus state: current_term: 1 leader_uuid: "4978301f9fbe4a53b8f556f31db457eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4978301f9fbe4a53b8f556f31db457eb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35319 } } }
I20260927 04:59:31.816226 26583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4978301f9fbe4a53b8f556f31db457eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4978301f9fbe4a53b8f556f31db457eb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35319 } } }
I20260927 04:59:31.816731 26583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:31.816731 26584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:31.819221 26591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:31.823923 26591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:31.832007 26591 catalog_manager.cc:1347] Generated new cluster ID: 88b8bc5e298443f18a223f7c88f7ef0e
I20260927 04:59:31.832124 26591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:31.846706 26591 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:31.847891 26591 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:31.859107 26591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4978301f9fbe4a53b8f556f31db457eb: Generated new TSK 0
I20260927 04:59:31.859905 26591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 04:59:31.896241 26608 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 04:59:31.896376 26608 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 04:59:31.897712 26607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:31.877183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35319 (local address 127.0.0.1:33762)
0927 04:59:31.877796 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:31.877812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:31.877935 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:31.878427 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:31.878430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:31.878441 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:59:31.878677 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:31.878685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.880159 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.880163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:31.881425 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:31.881438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.882467 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.882471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:31.882614 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:31.882647 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:59:31.897204 (+ 14557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 04:59:31.897417 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.169.126:35319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":338,"threads_started":1}
I20260927 04:59:31.897730 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:31.877321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.20.169.126:35319)
0927 04:59:31.877530 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:31.877537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:31.877898 (+ 361us) server_negotiation.cc:407] Connection header received
0927 04:59:31.878124 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:31.878130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:31.878222 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:31.878404 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:59:31.878832 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.880010 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.881571 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:31.882297 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:31.882437 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:31.882938 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:59:31.896977 (+ 14039us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 04:59:31.897402 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 04:59:31.897850 26580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 04:59:31.899288 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26510
I20260927 04:59:31.900559 26572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:31.992647 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26510
2026-09-27T04:59:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 04:59:32.049870 21157 mini_oidc.cc:152] Starting JWKS server
I20260927 04:59:32.051702 21157 webserver.cc:460] Webserver started at https://127.0.0.1:41307/ using document root <none> and password file <none>
I20260927 04:59:32.051942 21157 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 04:59:32.052577 21157 webserver.cc:460] Webserver started at http://127.0.0.1:39645/ using document root <none> and password file <none>
2026-09-27T04:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:32Z Disabled control of system clock
I20260927 04:59:32.080139 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45471
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:39743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45471
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 04:59:32.400075 26615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:32.400434 26615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:32.400588 26615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:32.407771 26615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:32.407902 26615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:32.407999 26615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:32.408107 26615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:32.408191 26615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:59:32.408277 26615 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:59:32.408356 26615 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41307/jwks.json
I20260927 04:59:32.418282 26615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:32.419577 26615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:32.421373 26615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:32.433859 26620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:32.433831 26621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:32.433888 26623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:32.434929 26615 server_base.cc:1034] running on GCE node
I20260927 04:59:32.435663 26615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:32.437134 26615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:32.438335 26615 hybrid_clock.cc:648] HybridClock initialized: now 1790485172438293 us; error 43 us; skew 500 ppm
I20260927 04:59:32.438668 26615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:32.441046 26615 webserver.cc:460] Webserver started at http://127.20.169.126:38141/ using document root <none> and password file <none>
I20260927 04:59:32.441711 26615 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:32.441797 26615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:32.442119 26615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:32.444934 26615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "10f7a1942d84464e87c61792ee2b75fb"
format_stamp: "Formatted at 2026-09-27 04:59:32 on dist-test-slave-39f2"
server_key: "84df89fe157787c0d2247ed236bc5bb0"
server_key_iv: "c16d18136a1959807b2002a88b90df51"
server_key_version: "encryptionkey@0"
I20260927 04:59:32.445713 26615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "10f7a1942d84464e87c61792ee2b75fb"
format_stamp: "Formatted at 2026-09-27 04:59:32 on dist-test-slave-39f2"
server_key: "84df89fe157787c0d2247ed236bc5bb0"
server_key_iv: "c16d18136a1959807b2002a88b90df51"
server_key_version: "encryptionkey@0"
I20260927 04:59:32.450636 26615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 04:59:32.453976 26629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:32.455221 26615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 04:59:32.455392 26615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "10f7a1942d84464e87c61792ee2b75fb"
format_stamp: "Formatted at 2026-09-27 04:59:32 on dist-test-slave-39f2"
server_key: "84df89fe157787c0d2247ed236bc5bb0"
server_key_iv: "c16d18136a1959807b2002a88b90df51"
server_key_version: "encryptionkey@0"
I20260927 04:59:32.455610 26615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:32.483043 26615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:32.483934 26615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:32.484256 26615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:32.502799 26615 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45471
I20260927 04:59:32.502820 26680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45471 every 8 connection(s)
I20260927 04:59:32.504132 26615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:32.508068 26681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:32.508241 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26615
I20260927 04:59:32.508435 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:32.509153 21157 external_mini_cluster.cc:1428] Setting key aef5a3d43f5dadeaf80e54f81c96719a
I20260927 04:59:32.520279 26681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb: Bootstrap starting.
I20260927 04:59:32.522547 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:32.511046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.20.169.126:45471)
0927 04:59:32.512055 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:32.512069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:32.512110 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:59:32.514205 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:32.514229 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:32.514692 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:32.515081 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:32.515629 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:32.516922 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:32.518223 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:32.518879 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:32.519087 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:32.520234 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:32.520242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:32.520255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:32.520892 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:32.520969 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:32.520989 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:32.521112 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:32.521356 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 04:59:32.521638 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":369,"threads_started":1}
I20260927 04:59:32.524036 26681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:32.525236 26681 log.cc:824] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:32.526659 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:59:32.528595 26681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb: No bootstrap required, opened a new log
I20260927 04:59:32.536260 26681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f7a1942d84464e87c61792ee2b75fb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45471 } }
I20260927 04:59:32.536599 26681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:32.536670 26681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f7a1942d84464e87c61792ee2b75fb, State: Initialized, Role: FOLLOWER
I20260927 04:59:32.537202 26681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f7a1942d84464e87c61792ee2b75fb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45471 } }
I20260927 04:59:32.537403 26681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:32.537499 26681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:32.537617 26681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:32.539944 26681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f7a1942d84464e87c61792ee2b75fb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45471 } }
I20260927 04:59:32.540408 26681 leader_election.cc:302] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10f7a1942d84464e87c61792ee2b75fb; no voters:
I20260927 04:59:32.540967 26681 leader_election.cc:288] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:32.541122 26686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:32.541833 26686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [term 1 LEADER]: Becoming Leader. State: Replica: 10f7a1942d84464e87c61792ee2b75fb, State: Running, Role: LEADER
I20260927 04:59:32.542397 26686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f7a1942d84464e87c61792ee2b75fb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45471 } }
I20260927 04:59:32.542982 26681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:32.545346 26687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10f7a1942d84464e87c61792ee2b75fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f7a1942d84464e87c61792ee2b75fb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45471 } } }
I20260927 04:59:32.545279 26688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f7a1942d84464e87c61792ee2b75fb. Latest consensus state: current_term: 1 leader_uuid: "10f7a1942d84464e87c61792ee2b75fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f7a1942d84464e87c61792ee2b75fb" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45471 } } }
I20260927 04:59:32.545773 26687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:32.545858 26688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:32.548379 26695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:32.553673 26695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:32.563823 26695 catalog_manager.cc:1347] Generated new cluster ID: e2ae79ae0eb74cb7a5e24902847f7891
I20260927 04:59:32.563967 26695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:32.579064 26695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:32.580371 26695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:32.591073 26695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10f7a1942d84464e87c61792ee2b75fb: Generated new TSK 0
I20260927 04:59:32.591922 26695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:59:32.611479 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:32.603177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.20.169.126:45471)
0927 04:59:32.603417 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:32.603427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:32.603890 (+ 463us) server_negotiation.cc:407] Connection header received
0927 04:59:32.604137 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:32.604142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:32.604219 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:32.604368 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:59:32.604996 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:32.605836 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:32.607776 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:32.608224 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:32.608358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:32.608962 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:59:32.610829 (+ 1867us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 04:59:32.611151 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 04:59:32.611646 26710 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:32.603170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45471 (local address 127.0.0.1:56738)
0927 04:59:32.603756 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:32.603776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:32.603938 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:32.604423 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:32.604429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:32.604443 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:59:32.604788 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:32.604801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:32.605978 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:32.605983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:32.607629 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:32.607644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:32.608410 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:32.608417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:32.608603 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:32.608647 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:59:32.611015 (+ 2368us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 04:59:32.611246 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.169.126:45471: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":312,"threads_started":1}
W20260927 04:59:32.611971 26684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 04:59:32.613793 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26615
I20260927 04:59:32.615165 26676 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 04:59:32.624862 26711 webserver.cc:533] Webserver: sslize failed:
E20260927 04:59:32.624997 26711 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 04:59:32.712635 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26615
2026-09-27T04:59:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T04:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:32Z Disabled control of system clock
I20260927 04:59:32.796147 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45771
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45771
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/testchainca.cert with env {}
W20260927 04:59:33.116782 26716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:33.117091 26716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:33.117178 26716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:33.124334 26716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:33.124444 26716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:33.124480 26716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:33.124538 26716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:33.124580 26716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:59:33.124639 26716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:59:33.124677 26716 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 04:59:33.134557 26716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:33.135833 26716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:33.137104 26716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:33.150821 26722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:33.150731 26724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:33.150861 26721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:33.151751 26716 server_base.cc:1034] running on GCE node
I20260927 04:59:33.152590 26716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:33.154207 26716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:33.155443 26716 hybrid_clock.cc:648] HybridClock initialized: now 1790485173155398 us; error 39 us; skew 500 ppm
I20260927 04:59:33.155823 26716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 04:59:33.158272 26716 webserver.cc:460] Webserver started at http://127.20.169.126:41371/ using document root <none> and password file <none>
I20260927 04:59:33.158993 26716 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:33.159128 26716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:33.159440 26716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:33.162457 26716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "9c7d82f222c94b359ebb053d3a02efdd"
format_stamp: "Formatted at 2026-09-27 04:59:33 on dist-test-slave-39f2"
server_key: "797374f7d0b08e113318a39352d8aef4"
server_key_iv: "87f67e4ba1f5e7afe026c6a5e87dc378"
server_key_version: "encryptionkey@0"
I20260927 04:59:33.163241 26716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "9c7d82f222c94b359ebb053d3a02efdd"
format_stamp: "Formatted at 2026-09-27 04:59:33 on dist-test-slave-39f2"
server_key: "797374f7d0b08e113318a39352d8aef4"
server_key_iv: "87f67e4ba1f5e7afe026c6a5e87dc378"
server_key_version: "encryptionkey@0"
I20260927 04:59:33.168946 26716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 04:59:33.172284 26730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:33.173760 26716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:33.173957 26716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "9c7d82f222c94b359ebb053d3a02efdd"
format_stamp: "Formatted at 2026-09-27 04:59:33 on dist-test-slave-39f2"
server_key: "797374f7d0b08e113318a39352d8aef4"
server_key_iv: "87f67e4ba1f5e7afe026c6a5e87dc378"
server_key_version: "encryptionkey@0"
I20260927 04:59:33.174170 26716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:33.198655 26716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:33.199434 26716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:33.199724 26716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:33.218691 26716 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45771
I20260927 04:59:33.218727 26782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45771 every 8 connection(s)
I20260927 04:59:33.220098 26716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:33.223869 26783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:33.224761 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26716
I20260927 04:59:33.225016 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:33.225646 21157 external_mini_cluster.cc:1428] Setting key 53595eddfa9aa43b193289b978f284de
I20260927 04:59:33.237393 26783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd: Bootstrap starting.
I20260927 04:59:33.239506 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:33.227625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.20.169.126:45771)
0927 04:59:33.228615 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:33.228633 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:33.228672 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:59:33.230256 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:33.230280 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:33.230718 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:33.231042 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:33.231588 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:33.232913 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:33.234747 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:33.235642 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:33.235819 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:33.237142 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:33.237150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:33.237165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:59:33.237763 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:33.237864 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:59:33.237885 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:33.237973 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:33.238203 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 04:59:33.238560 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":371,"threads_started":1}
I20260927 04:59:33.240969 26783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:33.242092 26783 log.cc:824] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:33.243558 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:59:33.245407 26783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd: No bootstrap required, opened a new log
I20260927 04:59:33.253046 26783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d82f222c94b359ebb053d3a02efdd" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45771 } }
I20260927 04:59:33.253357 26783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:33.253427 26783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7d82f222c94b359ebb053d3a02efdd, State: Initialized, Role: FOLLOWER
I20260927 04:59:33.253942 26783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d82f222c94b359ebb053d3a02efdd" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45771 } }
I20260927 04:59:33.254134 26783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:33.254231 26783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:33.254349 26783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:33.256405 26783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d82f222c94b359ebb053d3a02efdd" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45771 } }
I20260927 04:59:33.257028 26783 leader_election.cc:302] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c7d82f222c94b359ebb053d3a02efdd; no voters:
I20260927 04:59:33.257618 26783 leader_election.cc:288] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:33.257808 26788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:33.258584 26788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [term 1 LEADER]: Becoming Leader. State: Replica: 9c7d82f222c94b359ebb053d3a02efdd, State: Running, Role: LEADER
I20260927 04:59:33.259119 26788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d82f222c94b359ebb053d3a02efdd" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45771 } }
I20260927 04:59:33.259577 26783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:33.262041 26790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c7d82f222c94b359ebb053d3a02efdd. Latest consensus state: current_term: 1 leader_uuid: "9c7d82f222c94b359ebb053d3a02efdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d82f222c94b359ebb053d3a02efdd" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45771 } } }
I20260927 04:59:33.262444 26790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:33.262917 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c7d82f222c94b359ebb053d3a02efdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c7d82f222c94b359ebb053d3a02efdd" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45771 } } }
I20260927 04:59:33.263257 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:33.263423 26797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:33.269001 26797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:33.278380 26797 catalog_manager.cc:1347] Generated new cluster ID: a4bcf1cff5b84329abcc884f3e471479
I20260927 04:59:33.278519 26797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:33.299289 26797 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:33.300628 26797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:33.307541 26797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c7d82f222c94b359ebb053d3a02efdd: Generated new TSK 0
I20260927 04:59:33.308296 26797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:59:33.401227 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:33.325830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.20.169.126:45771)
0927 04:59:33.326086 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:33.326092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:33.326392 (+ 300us) server_negotiation.cc:407] Connection header received
0927 04:59:33.326611 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:33.326615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:33.326681 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:33.326791 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:59:33.327331 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:33.328042 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:33.330101 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:33.330551 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:33.330641 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:33.331188 (+ 547us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:59:33.400485 (+ 69297us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 04:59:33.400939 (+ 454us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59740: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 04:59:33.401400 26786 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59740: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 04:59:33.401611 26812 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:33.325650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:45771 (local address 127.0.0.1:59740)
0927 04:59:33.326287 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:33.326308 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:33.326448 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:33.326844 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:33.326849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:33.326865 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:59:33.327188 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:33.327199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:33.328183 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:33.328189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:33.329956 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:33.329972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:33.330708 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:33.330715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:33.330912 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:33.330956 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:59:33.400845 (+ 69889us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 04:59:33.401199 (+ 354us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.169.126:45771: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":328,"threads_started":1}
I20260927 04:59:33.403070 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26716
I20260927 04:59:33.404276 26778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:33.504282 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26716
2026-09-27T04:59:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (772 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (411 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:34 dist-test-slave-39f2 krb5kdc[26827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:34 dist-test-slave-39f2 krb5kdc[26827](info): set up 2 sockets
Sep 27 04:59:34 dist-test-slave-39f2 krb5kdc[26827](info): commencing operation
krb5kdc: starting...
W20260927 04:59:35.421036 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:35 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:35.636336 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:40985
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:40985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:35.959889 26840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:35.960240 26840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:35.960404 26840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:35.967619 26840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:35.967759 26840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:35.967877 26840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:35.967975 26840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:35.968060 26840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:35.977643 26840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:40985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:40985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:35.978991 26840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:35.980336 26840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:35.992266 26848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:35.992357 26846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:35.992398 26845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:35.993252 26840 server_base.cc:1034] running on GCE node
I20260927 04:59:35.994091 26840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:35.995627 26840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:35.996937 26840 hybrid_clock.cc:648] HybridClock initialized: now 1790485175996861 us; error 45 us; skew 500 ppm
I20260927 04:59:35.997315 26840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:35 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485175, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:36.000468 26840 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:36.002039 26840 webserver.cc:460] Webserver started at http://127.20.169.126:45877/ using document root <none> and password file <none>
I20260927 04:59:36.002731 26840 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:36.002836 26840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:36.003201 26840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:36.006242 26840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "60e0a9df95aa4f2c8413d4c4c1f626be"
format_stamp: "Formatted at 2026-09-27 04:59:35 on dist-test-slave-39f2"
server_key: "1500410ce58168123f5bdcf89706e80a"
server_key_iv: "d64b580743baaa1820a77a29498af3ae"
server_key_version: "encryptionkey@0"
I20260927 04:59:36.007071 26840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "60e0a9df95aa4f2c8413d4c4c1f626be"
format_stamp: "Formatted at 2026-09-27 04:59:35 on dist-test-slave-39f2"
server_key: "1500410ce58168123f5bdcf89706e80a"
server_key_iv: "d64b580743baaa1820a77a29498af3ae"
server_key_version: "encryptionkey@0"
I20260927 04:59:36.013871 26840 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 04:59:36.018435 26855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:36.019852 26840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 04:59:36.020009 26840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "60e0a9df95aa4f2c8413d4c4c1f626be"
format_stamp: "Formatted at 2026-09-27 04:59:35 on dist-test-slave-39f2"
server_key: "1500410ce58168123f5bdcf89706e80a"
server_key_iv: "d64b580743baaa1820a77a29498af3ae"
server_key_version: "encryptionkey@0"
I20260927 04:59:36.020176 26840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:36.031710 26840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:36.035815 26840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:36.036067 26840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:36.053997 26840 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:40985
I20260927 04:59:36.054032 26907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:40985 every 8 connection(s)
I20260927 04:59:36.055454 26840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:36.055851 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26840
I20260927 04:59:36.056097 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:36.057036 21157 external_mini_cluster.cc:1428] Setting key 3f2a6b26cfab42381571f6d2bd2cc220
I20260927 04:59:36.060700 26908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 04:59:36 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:36.074712 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be: Bootstrap starting.
I20260927 04:59:36.079325 26908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:36.080427 26908 log.cc:824] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:36.082939 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:36.059155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.20.169.126:40985)
0927 04:59:36.059967 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:36.059982 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:36.060035 (+ 53us) server_negotiation.cc:407] Connection header received
0927 04:59:36.062591 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:36.062620 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:36.063150 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:36.063626 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:36.065039 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:36.067371 (+ 2332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:36.068620 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:36.069757 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:36.069950 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:36.074438 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:36.074450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:36.074470 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:36.074518 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:36.078847 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:36.079596 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:36.079604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:36.079610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:36.079786 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:36.080170 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:36.080175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:36.080180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:36.080744 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:36.081083 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:36.081558 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 04:59:36.082026 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":319,"threads_started":1}
I20260927 04:59:36.083519 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be: No bootstrap required, opened a new log
I20260927 04:59:36.092351 26908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40985 } }
I20260927 04:59:36.092788 26908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:36.092974 26908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e0a9df95aa4f2c8413d4c4c1f626be, State: Initialized, Role: FOLLOWER
I20260927 04:59:36.093554 26908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40985 } }
I20260927 04:59:36.093804 26908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:36.093962 26908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:36.094157 26908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:36.096441 26908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40985 } }
I20260927 04:59:36.097004 26908 leader_election.cc:302] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e0a9df95aa4f2c8413d4c4c1f626be; no voters:
I20260927 04:59:36.097644 26908 leader_election.cc:288] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:36.097733 26913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:36.098485 26913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [term 1 LEADER]: Becoming Leader. State: Replica: 60e0a9df95aa4f2c8413d4c4c1f626be, State: Running, Role: LEADER
I20260927 04:59:36.099112 26913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40985 } }
I20260927 04:59:36.100066 26908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:36.102339 26914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40985 } } }
I20260927 04:59:36.102329 26915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e0a9df95aa4f2c8413d4c4c1f626be. Latest consensus state: current_term: 1 leader_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0a9df95aa4f2c8413d4c4c1f626be" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40985 } } }
I20260927 04:59:36.102787 26914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:36.102790 26915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:36.106031 26919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:36.112102 26919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:36.121537 26919 catalog_manager.cc:1347] Generated new cluster ID: d80d92cf9aee40a69402e333c77ed075
I20260927 04:59:36.121652 26919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:36.131700 26919 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:36.133459 26919 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:36.141537 26919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e0a9df95aa4f2c8413d4c4c1f626be: Generated new TSK 0
I20260927 04:59:36.142524 26919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:36.202688 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:40985
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:36.528576 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:36.528921 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:36.529080 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:36.536180 26934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:36.536324 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:36.536509 26934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:36.546128 26934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:40985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:36.547403 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:36.548686 26934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:36.561512 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:36.561478 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:36.561703 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:36.562777 26934 server_base.cc:1034] running on GCE node
I20260927 04:59:36.563484 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:36.564504 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:36.565759 26934 hybrid_clock.cc:648] HybridClock initialized: now 1790485176565711 us; error 45 us; skew 500 ppm
I20260927 04:59:36.566131 26934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:36 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485176, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:36.569315 26934 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:36.571622 26934 webserver.cc:460] Webserver started at http://127.20.169.65:34257/ using document root <none> and password file <none>
I20260927 04:59:36.572588 26934 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:36.572726 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:36.573069 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:36.576041 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "1b1da8b0b97c4c17805a295a337f108a"
format_stamp: "Formatted at 2026-09-27 04:59:36 on dist-test-slave-39f2"
server_key: "0b509bdc6e41a4d684e9d55263808a29"
server_key_iv: "31b625b9b16bfd4560d56e1ff76c9789"
server_key_version: "encryptionkey@0"
I20260927 04:59:36.576880 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1da8b0b97c4c17805a295a337f108a"
format_stamp: "Formatted at 2026-09-27 04:59:36 on dist-test-slave-39f2"
server_key: "0b509bdc6e41a4d684e9d55263808a29"
server_key_iv: "31b625b9b16bfd4560d56e1ff76c9789"
server_key_version: "encryptionkey@0"
I20260927 04:59:36.582203 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 04:59:36.586067 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:36.587476 26934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:36.587626 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "1b1da8b0b97c4c17805a295a337f108a"
format_stamp: "Formatted at 2026-09-27 04:59:36 on dist-test-slave-39f2"
server_key: "0b509bdc6e41a4d684e9d55263808a29"
server_key_iv: "31b625b9b16bfd4560d56e1ff76c9789"
server_key_version: "encryptionkey@0"
I20260927 04:59:36.587796 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:36.600471 26934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:36.604606 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:36.604890 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:36.605758 26934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:36.607084 26934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:36.607170 26934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:36.607306 26934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:36.607352 26934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:36.641176 26934 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:42779
I20260927 04:59:36.641198 27062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:42779 every 8 connection(s)
I20260927 04:59:36.642514 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:36.645259 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 26934
I20260927 04:59:36.645529 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:36.646234 21157 external_mini_cluster.cc:1428] Setting key 217ab1f6446b8efcaec3ff7849aaa003
Sep 27 04:59:36 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485176, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:36.662926 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:36.645260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:56389 (local address 127.20.169.126:40985)
0927 04:59:36.645844 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:36.645852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:36.646835 (+ 983us) server_negotiation.cc:407] Connection header received
0927 04:59:36.648897 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:36.648902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:36.649021 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:36.649136 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:36.650579 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:36.651285 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:36.652846 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:36.653231 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:36.653384 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:36.657255 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:36.657260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:36.657271 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:36.657303 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:36.659847 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:36.660562 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:36.660569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:36.660572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:36.660706 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:36.661163 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:36.661171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:36.661174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:36.661471 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:36.661633 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:36.662325 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 04:59:36.662567 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":248,"threads_started":1}
I20260927 04:59:36.663888 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:36.645835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:40985 (local address 127.20.169.65:56389)
0927 04:59:36.646640 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:36.646692 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:36.648608 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:36.649374 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:36.649382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:36.649855 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:36.650421 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:36.650438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:36.651440 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:36.651445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:36.652686 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:36.652697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:36.653452 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:36.653459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:36.653715 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:36.654403 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:36.654424 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:36.657043 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:36.659997 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:36.660007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:36.660022 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:36.660443 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:36.660857 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:36.660863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:36.660866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:36.661050 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:36.661602 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:36.661610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:36.661955 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 04:59:36.662220 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 04:59:36.662745 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":337,"threads_started":1}
I20260927 04:59:36.665647 27063 heartbeater.cc:347] Connected to a master server at 127.20.169.126:40985
I20260927 04:59:36.665961 27063 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 04:59:36.666713 27063 heartbeater.cc:511] Master 127.20.169.126:40985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 04:59:36.669143 26872 ts_manager.cc:194] Registered new tserver with Master: 1b1da8b0b97c4c17805a295a337f108a (127.20.169.65:42779)
I20260927 04:59:36.670836 26872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:56389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:36.693557 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:40985
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:37.030270 27069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:37.030678 27069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:37.030843 27069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:37.042290 27069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:37.042451 27069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:37.042649 27069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:37.052433 27069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:40985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:37.053858 27069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:37.055151 27069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:37.067978 27075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:37.068146 27074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:37.068984 27077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:37.070133 27069 server_base.cc:1034] running on GCE node
I20260927 04:59:37.070763 27069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:37.071736 27069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:37.072919 27069 hybrid_clock.cc:648] HybridClock initialized: now 1790485177072863 us; error 72 us; skew 500 ppm
I20260927 04:59:37.073230 27069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.076277 27069 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:37.077793 27069 webserver.cc:460] Webserver started at http://127.20.169.66:34735/ using document root <none> and password file <none>
I20260927 04:59:37.078496 27069 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:37.078640 27069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:37.078948 27069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:37.081880 27069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "aa8f9c0c257d413188d1bbc054ce2cf8"
format_stamp: "Formatted at 2026-09-27 04:59:37 on dist-test-slave-39f2"
server_key: "ed9d5383d4c76822c56f8598b8cdfc7f"
server_key_iv: "583f1fcafa32285c7b1ae1c5c5258b82"
server_key_version: "encryptionkey@0"
I20260927 04:59:37.082700 27069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "aa8f9c0c257d413188d1bbc054ce2cf8"
format_stamp: "Formatted at 2026-09-27 04:59:37 on dist-test-slave-39f2"
server_key: "ed9d5383d4c76822c56f8598b8cdfc7f"
server_key_iv: "583f1fcafa32285c7b1ae1c5c5258b82"
server_key_version: "encryptionkey@0"
I20260927 04:59:37.087752 27069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:59:37.091163 27084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:37.092855 27069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:37.093047 27069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "aa8f9c0c257d413188d1bbc054ce2cf8"
format_stamp: "Formatted at 2026-09-27 04:59:37 on dist-test-slave-39f2"
server_key: "ed9d5383d4c76822c56f8598b8cdfc7f"
server_key_iv: "583f1fcafa32285c7b1ae1c5c5258b82"
server_key_version: "encryptionkey@0"
I20260927 04:59:37.093254 27069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:37.116523 27069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:37.120656 27069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:37.121002 27069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:37.122072 27069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:37.123468 27069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:37.123572 27069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:37.123728 27069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:37.123819 27069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:37.156242 27069 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:42939
I20260927 04:59:37.156327 27196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:42939 every 8 connection(s)
I20260927 04:59:37.157624 27069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:37.163972 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27069
I20260927 04:59:37.164182 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:37.164759 21157 external_mini_cluster.cc:1428] Setting key c7b779a9feed4208ef45afb292e7d655
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:37.178392 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.160287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:48377 (local address 127.20.169.126:40985)
0927 04:59:37.160499 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.160504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.161758 (+ 1254us) server_negotiation.cc:407] Connection header received
0927 04:59:37.163633 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.163640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.163732 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.163865 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.165760 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.166806 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.168388 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.168870 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.169003 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.172492 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.172497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.172500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.172530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.175079 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.176005 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.176008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.176011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.176130 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.176607 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.176610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.176612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.176927 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.177034 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.177977 (+ 943us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.178148 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 04:59:37.179845 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.160729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:40985 (local address 127.20.169.66:48377)
0927 04:59:37.161576 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:37.161621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:37.163316 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:37.164143 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:37.164157 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:37.164741 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:37.165542 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:37.165571 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.166955 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.166960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:37.168221 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:37.168233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.169106 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.169113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:37.169343 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.170042 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:37.170065 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:37.172163 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:37.175215 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:37.175226 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:37.175247 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:37.175888 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:37.176244 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:37.176252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:37.176258 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:37.176503 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:37.177061 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:37.177071 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:37.177524 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 04:59:37.177844 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 04:59:37.178429 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":299,"threads_started":1}
I20260927 04:59:37.181310 27197 heartbeater.cc:347] Connected to a master server at 127.20.169.126:40985
I20260927 04:59:37.181654 27197 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:37.182427 27197 heartbeater.cc:511] Master 127.20.169.126:40985 requested a full tablet report, sending...
I20260927 04:59:37.183992 26872 ts_manager.cc:194] Registered new tserver with Master: aa8f9c0c257d413188d1bbc054ce2cf8 (127.20.169.66:42939)
I20260927 04:59:37.184787 26872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:48377
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:37.212678 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:40985
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:37.528224 27202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:37.528506 27202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:37.528666 27202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:37.535988 27202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:37.536089 27202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:37.536260 27202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:37.546357 27202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:40985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:37.547693 27202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:37.549054 27202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:37.562484 27208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:37.562579 27207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:37.562558 27210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:37.563519 27202 server_base.cc:1034] running on GCE node
I20260927 04:59:37.564177 27202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:37.565248 27202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:37.566475 27202 hybrid_clock.cc:648] HybridClock initialized: now 1790485177566429 us; error 37 us; skew 500 ppm
I20260927 04:59:37.566900 27202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.570395 27202 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:37.571858 27202 webserver.cc:460] Webserver started at http://127.20.169.67:37991/ using document root <none> and password file <none>
I20260927 04:59:37.572500 27202 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:37.572606 27202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:37.573009 27202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:37.577024 27202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "eb69834011db4c86b1691eb795eecafd"
format_stamp: "Formatted at 2026-09-27 04:59:37 on dist-test-slave-39f2"
server_key: "858272bbb04c47f357d2c6ffe123ab8c"
server_key_iv: "e1a212b23ff299cffc3787e12837e2ee"
server_key_version: "encryptionkey@0"
I20260927 04:59:37.577863 27202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "eb69834011db4c86b1691eb795eecafd"
format_stamp: "Formatted at 2026-09-27 04:59:37 on dist-test-slave-39f2"
server_key: "858272bbb04c47f357d2c6ffe123ab8c"
server_key_iv: "e1a212b23ff299cffc3787e12837e2ee"
server_key_version: "encryptionkey@0"
I20260927 04:59:37.583123 27202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 04:59:37.586732 27217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:37.588057 27202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:37.588245 27202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "eb69834011db4c86b1691eb795eecafd"
format_stamp: "Formatted at 2026-09-27 04:59:37 on dist-test-slave-39f2"
server_key: "858272bbb04c47f357d2c6ffe123ab8c"
server_key_iv: "e1a212b23ff299cffc3787e12837e2ee"
server_key_version: "encryptionkey@0"
I20260927 04:59:37.588469 27202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:37.608620 27202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:37.613008 27202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:37.613294 27202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:37.614236 27202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:37.615609 27202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:37.615715 27202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:37.615865 27202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:37.615957 27202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:37.650544 27202 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:35341
I20260927 04:59:37.650601 27329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:35341 every 8 connection(s)
I20260927 04:59:37.651785 27202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:37.661943 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27202
I20260927 04:59:37.662340 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:37.662942 21157 external_mini_cluster.cc:1428] Setting key afa858919a666dd97df8ecd5cb0981a6
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:37.673362 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.654652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56273 (local address 127.20.169.126:40985)
0927 04:59:37.654837 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.654841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.656056 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 04:59:37.657980 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.657985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.658054 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.658155 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.659620 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.660479 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.662257 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.662973 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.663137 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.666547 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.666552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.666556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.666592 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.669828 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.670912 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.670917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.670919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.671056 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.671558 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.671562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.671564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.671876 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.671991 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.672889 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.673094 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 04:59:37.673995 27063 heartbeater.cc:503] Master 127.20.169.126:40985 was elected leader, sending a full tablet report...
I20260927 04:59:37.674546 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.655153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:40985 (local address 127.20.169.67:56273)
0927 04:59:37.655882 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:37.655923 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:37.657630 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:37.658377 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:37.658385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:37.658879 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:37.659441 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:37.659457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.660636 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.660640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:37.661948 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:37.661960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.663212 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.663217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:37.663416 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.664105 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:37.664133 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:37.666349 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:37.669973 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:37.669980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:37.669995 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:37.670751 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:37.671165 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:37.671169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:37.671171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:37.671432 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:37.672006 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:37.672012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:37.672407 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 04:59:37.672655 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 04:59:37.673185 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260927 04:59:37.676384 27330 heartbeater.cc:347] Connected to a master server at 127.20.169.126:40985
I20260927 04:59:37.676755 27330 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:37.677873 27330 heartbeater.cc:511] Master 127.20.169.126:40985 requested a full tablet report, sending...
I20260927 04:59:37.679486 26871 ts_manager.cc:194] Registered new tserver with Master: eb69834011db4c86b1691eb795eecafd (127.20.169.67:35341)
I20260927 04:59:37.680301 26871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:56273
I20260927 04:59:37.690877 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/test-admin.keytab.
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.718669 21157 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 04:59:37.723832 27338 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:37.721412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.20.169.65:42779)
0927 04:59:37.722075 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.722081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.722128 (+ 47us) server_negotiation.cc:407] Connection header received
0927 04:59:37.722204 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.722210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.722417 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.722548 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.723471 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47866: BlockingRecv error: recv got EOF from 127.0.0.1:47866 (error 108)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":363,"threads_started":1}
W20260927 04:59:37.724319 21157 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc' of type FILE
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.726207 21157 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.727517 21157 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:37.742605 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.729015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.20.169.65:42779)
0927 04:59:37.729160 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.729164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.729577 (+ 413us) server_negotiation.cc:407] Connection header received
0927 04:59:37.729725 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.729729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.729783 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.729859 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.730948 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.731892 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.733299 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.733836 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.733929 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.737636 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.737646 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.737660 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.737692 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.740328 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.741021 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.741028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.741030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.741140 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.741460 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.741463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.741465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.741774 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.742004 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.742183 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.742367 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 04:59:37.748431 27338 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:37.746715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.20.169.65:42779)
0927 04:59:37.746905 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.746911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.747096 (+ 185us) server_negotiation.cc:407] Connection header received
0927 04:59:37.747282 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.747286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.747345 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.747440 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.748164 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47882: BlockingRecv error: recv got EOF from 127.0.0.1:47882 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 04:59:37.748606 21157 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc' of type FILE
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.750154 21157 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.751471 21157 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:37.765486 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.752916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.20.169.65:42779)
0927 04:59:37.753136 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.753143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.753366 (+ 223us) server_negotiation.cc:407] Connection header received
0927 04:59:37.753554 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.753559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.753617 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.753711 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.754568 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.755245 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.756862 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.757289 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.757420 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.760856 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.760862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.760867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.760904 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.763176 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.763767 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.763774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.763777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.763892 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.764253 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.764258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.764262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.764507 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.764761 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.765048 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.765247 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 04:59:37.769831 27338 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:37.768112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.20.169.65:42779)
0927 04:59:37.768276 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.768282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.768591 (+ 309us) server_negotiation.cc:407] Connection header received
0927 04:59:37.768764 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.768775 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.768901 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.769003 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.769628 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47900: BlockingRecv error: recv got EOF from 127.0.0.1:47900 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 04:59:37.770236 21157 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485126081201-21157-0/krb5kdc/krb5cc)
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.771620 21157 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:37.772881 21157 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:37.786202 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.774019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.20.169.65:42779)
0927 04:59:37.774220 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.774224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.774471 (+ 247us) server_negotiation.cc:407] Connection header received
0927 04:59:37.774660 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.774664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.774708 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.774799 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.775650 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.776363 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.777591 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.778003 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.778130 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.781314 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.781319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.781322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.781349 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.783764 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.784360 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.784372 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.784375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.784491 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.784868 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.784873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.784876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.785130 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.785463 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.785719 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.785912 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 04:59:37.787763 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26934
I20260927 04:59:37.799480 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.788572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.20.169.65:42779)
0927 04:59:37.788747 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.788752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.788964 (+ 212us) server_negotiation.cc:407] Connection header received
0927 04:59:37.789081 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.789084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.789137 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.789214 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.790072 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.790759 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.792002 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.792477 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.792568 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.794547 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.794551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.794554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.794581 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.797218 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.797761 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.797765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.797767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.797907 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.798285 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.798289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.798292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.798588 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.798782 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.799046 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.799193 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 04:59:37.800120 27058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:37.885854 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26934
I20260927 04:59:37.903847 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27069
Sep 27 04:59:37 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:37.918732 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:37.904729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.20.169.66:42939)
0927 04:59:37.905373 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:37.905380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:37.905413 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:59:37.905476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:37.905483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:37.905693 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:37.905820 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:37.906828 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.907906 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.909167 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:37.909780 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:37.909879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:37.913618 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:37.913628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:37.913642 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:37.913673 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:37.916262 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.916959 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.916965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.916968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.917080 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:37.917385 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:37.917388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:37.917390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:37.917727 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:37.917953 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:37.918220 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 04:59:37.918439 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":353,"threads_started":1}
I20260927 04:59:37.919966 27192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:38.002682 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27069
I20260927 04:59:38.019296 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27202
Sep 27 04:59:38 dist-test-slave-39f2 krb5kdc[26827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:38.035194 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:38.020401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.20.169.67:35341)
0927 04:59:38.021125 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:38.021132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:38.021160 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:59:38.021223 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:38.021228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:38.021440 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:38.021568 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:38.022556 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:38.023576 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:38.024950 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:38.025537 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:38.025636 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:38.029599 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:38.029610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:38.029626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:38.029663 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:38.032376 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:38.033159 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:38.033168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:38.033170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:38.033290 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:38.033676 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:38.033682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:38.033685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:38.034035 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:38.034359 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:38.034636 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 04:59:38.034894 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":401,"threads_started":1}
I20260927 04:59:38.036569 27325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:38.119709 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27202
I20260927 04:59:38.136385 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 26840
I20260927 04:59:38.137738 26903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:38.240908 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 26840
2026-09-27T04:59:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3918 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:38 dist-test-slave-39f2 krb5kdc[27358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:38 dist-test-slave-39f2 krb5kdc[27358](info): set up 2 sockets
Sep 27 04:59:38 dist-test-slave-39f2 krb5kdc[27358](info): commencing operation
krb5kdc: starting...
W20260927 04:59:39.337314 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:39 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab.
Entry for principal oryx/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab.
I20260927 04:59:39.552184 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:36927
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab
--principal=oryx/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:39.879271 27371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:39.879602 27371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:39.879773 27371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:39.886979 27371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:39.887118 27371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:39.887226 27371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:39.887326 27371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:39.887414 27371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:39.897190 27371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab
--principal=oryx/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:39.898651 27371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:39.899962 27371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:39.912263 27377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:39.912235 27379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:39.912266 27376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:39.918399 27371 server_base.cc:1034] running on GCE node
I20260927 04:59:39.919225 27371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:39.920940 27371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:39.922145 27371 hybrid_clock.cc:648] HybridClock initialized: now 1790485179922080 us; error 63 us; skew 500 ppm
I20260927 04:59:39.922549 27371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:39 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:39.925740 27371 init.cc:377] Logged in from keytab as oryx/127.20.169.126@KRBTEST.COM (short username oryx)
I20260927 04:59:39.927312 27371 webserver.cc:460] Webserver started at http://127.20.169.126:35253/ using document root <none> and password file <none>
I20260927 04:59:39.928049 27371 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:39.928195 27371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:39.928506 27371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:39.931579 27371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "99fa63b96cd341f9b51f24e007bff59d"
format_stamp: "Formatted at 2026-09-27 04:59:39 on dist-test-slave-39f2"
server_key: "0d99d803647b9b8b0686b4f4e0f9efbb"
server_key_iv: "2cf7592b624381201462ffa69520522e"
server_key_version: "encryptionkey@0"
I20260927 04:59:39.932332 27371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "99fa63b96cd341f9b51f24e007bff59d"
format_stamp: "Formatted at 2026-09-27 04:59:39 on dist-test-slave-39f2"
server_key: "0d99d803647b9b8b0686b4f4e0f9efbb"
server_key_iv: "2cf7592b624381201462ffa69520522e"
server_key_version: "encryptionkey@0"
I20260927 04:59:39.937479 27371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 04:59:39.940711 27386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:39.942266 27371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 04:59:39.942456 27371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "99fa63b96cd341f9b51f24e007bff59d"
format_stamp: "Formatted at 2026-09-27 04:59:39 on dist-test-slave-39f2"
server_key: "0d99d803647b9b8b0686b4f4e0f9efbb"
server_key_iv: "2cf7592b624381201462ffa69520522e"
server_key_version: "encryptionkey@0"
I20260927 04:59:39.942647 27371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:39.969791 27371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:39.974171 27371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:39.974465 27371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:39.993129 27371 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:36927
I20260927 04:59:39.993209 27437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:36927 every 8 connection(s)
I20260927 04:59:39.994491 27371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:39.998590 27438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:40.002630 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27371
I20260927 04:59:40.002899 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:40.003808 21157 external_mini_cluster.cc:1428] Setting key 27b3f2294e51b1a12cac9edecad3c591
I20260927 04:59:40.013625 27438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Bootstrap starting.
I20260927 04:59:40.016619 27438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:40.017997 27438 log.cc:824] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Log is configured to *not* fsync() on all Append() calls
Sep 27 04:59:40 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:40.021040 27438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: No bootstrap required, opened a new log
I20260927 04:59:40.028527 27438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } }
I20260927 04:59:40.028918 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:40.006220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.20.169.126:36927)
0927 04:59:40.007050 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:40.007062 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:40.007107 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:59:40.009429 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:40.009454 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:40.009895 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:40.010277 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:40.012084 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:40.013678 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:40.015053 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:40.015781 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:40.015976 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:40.020971 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:40.020979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:40.020995 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:40.021036 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:40.024611 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:40.025360 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:40.025367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:40.025373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:40.025543 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:40.025970 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:40.025978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:40.025986 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:40.026523 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:40.026891 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:40.027350 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 04:59:40.027775 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":404,"threads_started":1}
I20260927 04:59:40.028973 27438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:40.029086 27438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fa63b96cd341f9b51f24e007bff59d, State: Initialized, Role: FOLLOWER
I20260927 04:59:40.029778 27438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } }
I20260927 04:59:40.030128 27438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:40.030305 27438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:40.030517 27438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:40.033281 27438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } }
I20260927 04:59:40.033787 27438 leader_election.cc:302] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99fa63b96cd341f9b51f24e007bff59d; no voters:
I20260927 04:59:40.034452 27438 leader_election.cc:288] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:40.034519 27443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:40.036116 27443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 LEADER]: Becoming Leader. State: Replica: 99fa63b96cd341f9b51f24e007bff59d, State: Running, Role: LEADER
I20260927 04:59:40.036607 27443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } }
I20260927 04:59:40.036948 27438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:40.039314 27444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } }
I20260927 04:59:40.039773 27444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:40.040412 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99fa63b96cd341f9b51f24e007bff59d. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } }
I20260927 04:59:40.040797 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:40.041383 27451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:40.047338 27451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:40.056288 27451 catalog_manager.cc:1347] Generated new cluster ID: 56befa11045343c0be45d783eb3564eb
I20260927 04:59:40.056399 27451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:40.065794 27451 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:40.067464 27451 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:40.074797 27451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Generated new TSK 0
I20260927 04:59:40.075709 27451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab.
Entry for principal oryx/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab.
I20260927 04:59:40.130200 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36927
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:40.471339 27464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:40.471614 27464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:40.471761 27464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:40.479056 27464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:40.479171 27464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:40.479346 27464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:40.489084 27464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:40.490399 27464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:40.491739 27464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:40.504897 27470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:40.505066 27472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:40.505430 27469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:40.506383 27464 server_base.cc:1034] running on GCE node
I20260927 04:59:40.507030 27464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:40.507998 27464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:40.509224 27464 hybrid_clock.cc:648] HybridClock initialized: now 1790485180509181 us; error 53 us; skew 500 ppm
I20260927 04:59:40.509589 27464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:40 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485180, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:40.512876 27464 init.cc:377] Logged in from keytab as oryx/127.20.169.65@KRBTEST.COM (short username oryx)
I20260927 04:59:40.514273 27464 webserver.cc:460] Webserver started at http://127.20.169.65:44817/ using document root <none> and password file <none>
I20260927 04:59:40.514973 27464 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:40.515121 27464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:40.515426 27464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:40.518288 27464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "f245a22d06014d6d9f47477d25dcd6fc"
format_stamp: "Formatted at 2026-09-27 04:59:40 on dist-test-slave-39f2"
server_key: "f22461234dd644742b2144fb61ae651d"
server_key_iv: "16aa6bc582e4884ebb266646747c806b"
server_key_version: "encryptionkey@0"
I20260927 04:59:40.519054 27464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "f245a22d06014d6d9f47477d25dcd6fc"
format_stamp: "Formatted at 2026-09-27 04:59:40 on dist-test-slave-39f2"
server_key: "f22461234dd644742b2144fb61ae651d"
server_key_iv: "16aa6bc582e4884ebb266646747c806b"
server_key_version: "encryptionkey@0"
I20260927 04:59:40.524021 27464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 04:59:40.527415 27479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:40.528883 27464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:40.529047 27464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "f245a22d06014d6d9f47477d25dcd6fc"
format_stamp: "Formatted at 2026-09-27 04:59:40 on dist-test-slave-39f2"
server_key: "f22461234dd644742b2144fb61ae651d"
server_key_iv: "16aa6bc582e4884ebb266646747c806b"
server_key_version: "encryptionkey@0"
I20260927 04:59:40.529241 27464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:40.550880 27464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:40.555630 27464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:40.555936 27464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:40.556957 27464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:40.558252 27464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:40.558359 27464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:40.558507 27464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:40.558584 27464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:40.593631 27464 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:41387
I20260927 04:59:40.593712 27592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:41387 every 8 connection(s)
I20260927 04:59:40.594935 27464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:40.601413 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27464
I20260927 04:59:40.601655 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:40.602294 21157 external_mini_cluster.cc:1428] Setting key d80e4b0967fc6e5e010b6ed14b844f37
Sep 27 04:59:40 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485180, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:40.616326 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:40.597510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:36881 (local address 127.20.169.126:36927)
0927 04:59:40.597963 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:40.597969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:40.598883 (+ 914us) server_negotiation.cc:407] Connection header received
0927 04:59:40.600858 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:40.600864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:40.600976 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:40.601096 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:40.602703 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:40.603604 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:40.605395 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:40.605804 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:40.605948 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:40.609691 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:40.609696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:40.609699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:40.609731 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:40.612771 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:40.613555 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:40.613562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:40.613565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:40.613701 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:40.614147 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:40.614154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:40.614157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:40.614450 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:40.614798 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:40.615622 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 04:59:40.615905 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260927 04:59:40.617436 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:40.597985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:36881)
0927 04:59:40.598718 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:40.598760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:40.600568 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:40.601338 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:40.601345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:40.601938 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:40.602540 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:40.602558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:40.603787 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:40.603793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:40.605228 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:40.605240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:40.606060 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:40.606067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:40.606295 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:40.606993 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:40.607015 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:40.609457 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:40.612973 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:40.612982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:40.613000 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:40.613429 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:40.613845 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:40.613849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:40.613851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:40.614034 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:40.614761 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:40.614770 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:40.615149 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 04:59:40.615517 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 04:59:40.616099 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":299,"threads_started":1}
I20260927 04:59:40.619206 27593 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36927
I20260927 04:59:40.619540 27593 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:40.620285 27593 heartbeater.cc:511] Master 127.20.169.126:36927 requested a full tablet report, sending...
I20260927 04:59:40.622742 27403 ts_manager.cc:194] Registered new tserver with Master: f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387)
WARNING: no policy specified for oryx/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 04:59:40.624997 27403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:36881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab.
Entry for principal oryx/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab.
I20260927 04:59:40.652562 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36927
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:40.973480 27599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:40.973735 27599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:40.973847 27599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:40.980944 27599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:40.981045 27599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:40.981199 27599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:40.990830 27599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:40.992177 27599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:40.993903 27599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:41.009238 27604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:41.009379 27607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:41.009425 27605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:41.010136 27599 server_base.cc:1034] running on GCE node
I20260927 04:59:41.010831 27599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:41.011983 27599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:41.013221 27599 hybrid_clock.cc:648] HybridClock initialized: now 1790485181013172 us; error 41 us; skew 500 ppm
I20260927 04:59:41.013545 27599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:41 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485181, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:41.016700 27599 init.cc:377] Logged in from keytab as oryx/127.20.169.66@KRBTEST.COM (short username oryx)
I20260927 04:59:41.018765 27599 webserver.cc:460] Webserver started at http://127.20.169.66:45891/ using document root <none> and password file <none>
I20260927 04:59:41.019757 27599 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:41.019904 27599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:41.020200 27599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:41.024039 27599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "ab7a27d683cb4d51884bfb576d629c97"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "de1f3e4ce44036c750e590bddf670595"
server_key_iv: "33601e7509f31c021a0ad20e4de26c86"
server_key_version: "encryptionkey@0"
I20260927 04:59:41.025187 27599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "ab7a27d683cb4d51884bfb576d629c97"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "de1f3e4ce44036c750e590bddf670595"
server_key_iv: "33601e7509f31c021a0ad20e4de26c86"
server_key_version: "encryptionkey@0"
I20260927 04:59:41.031057 27599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 04:59:41.034628 27614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:41.036062 27599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 04:59:41.036234 27599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "ab7a27d683cb4d51884bfb576d629c97"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "de1f3e4ce44036c750e590bddf670595"
server_key_iv: "33601e7509f31c021a0ad20e4de26c86"
server_key_version: "encryptionkey@0"
I20260927 04:59:41.036450 27599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:41.056567 27599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:41.060796 27599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:41.061142 27599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:41.062124 27599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:41.063400 27599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:41.063510 27599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:41.063649 27599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:41.063719 27599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:41.097266 27599 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:36259
I20260927 04:59:41.097349 27727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:36259 every 8 connection(s)
I20260927 04:59:41.098542 27599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:41.103770 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27599
I20260927 04:59:41.104003 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:41.104422 21157 external_mini_cluster.cc:1428] Setting key f4351466ce6a1ced7acfba97f54d2fbf
Sep 27 04:59:41 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485181, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:41.120486 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:41.101278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:35421 (local address 127.20.169.126:36927)
0927 04:59:41.101516 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:41.101521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:41.102613 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 04:59:41.104520 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:41.104526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:41.104616 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:41.104724 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:41.106428 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.107487 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.109430 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.110160 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.110307 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:41.114307 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:41.114311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:41.114315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:41.114353 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:41.117052 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:41.118082 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:41.118090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:41.118096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:41.118251 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:41.118731 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:41.118735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:41.118737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:41.118995 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:41.119112 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:41.120062 (+ 950us) server_negotiation.cc:299] Negotiation successful
0927 04:59:41.120236 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 04:59:41.122169 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:41.101768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:35421)
0927 04:59:41.102450 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:41.102486 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:41.104261 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:41.105011 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:41.105022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:41.105600 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:41.106239 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:41.106265 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.107682 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.107687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:41.109278 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:41.109292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.110416 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.110423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:41.110662 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:41.111431 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:41.111459 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:41.114072 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:41.117189 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:41.117201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:41.117223 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:41.117940 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:41.118366 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:41.118374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:41.118380 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:41.118622 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:41.119127 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:41.119138 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:41.119637 (+ 499us) client_negotiation.cc:769] Sending connection context
0927 04:59:41.119957 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 04:59:41.120587 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":264,"threads_started":1}
I20260927 04:59:41.123934 27728 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36927
I20260927 04:59:41.124244 27728 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:41.125056 27728 heartbeater.cc:511] Master 127.20.169.126:36927 requested a full tablet report, sending...
I20260927 04:59:41.126582 27403 ts_manager.cc:194] Registered new tserver with Master: ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
WARNING: no policy specified for oryx/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 04:59:41.127677 27403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:35421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab.
Entry for principal oryx/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab.
I20260927 04:59:41.156392 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36927
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:41.484215 27733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:41.484514 27733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:41.484709 27733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:41.492102 27733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:41.492216 27733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:41.492475 27733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:41.502605 27733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:41.503921 27733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:41.505394 27733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:41.519888 27738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:41.519917 27741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:41.519888 27739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:41.520700 27733 server_base.cc:1034] running on GCE node
I20260927 04:59:41.521452 27733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:41.522486 27733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:41.523747 27733 hybrid_clock.cc:648] HybridClock initialized: now 1790485181523698 us; error 68 us; skew 500 ppm
I20260927 04:59:41.524098 27733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:41 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485181, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:41.527585 27733 init.cc:377] Logged in from keytab as oryx/127.20.169.67@KRBTEST.COM (short username oryx)
I20260927 04:59:41.529291 27733 webserver.cc:460] Webserver started at http://127.20.169.67:42853/ using document root <none> and password file <none>
I20260927 04:59:41.530058 27733 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:41.530223 27733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:41.530562 27733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:41.534024 27733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "1769503332dc41ee90ab181cfec04f8f"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "a94254d8ca45fbe7186863da0d4ab08b"
server_key_iv: "b4b37e2439b1d514a650a4f894d48e91"
server_key_version: "encryptionkey@0"
I20260927 04:59:41.534883 27733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "1769503332dc41ee90ab181cfec04f8f"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "a94254d8ca45fbe7186863da0d4ab08b"
server_key_iv: "b4b37e2439b1d514a650a4f894d48e91"
server_key_version: "encryptionkey@0"
I20260927 04:59:41.540617 27733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 04:59:41.544516 27748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:41.545959 27733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:41.546181 27733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "1769503332dc41ee90ab181cfec04f8f"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "a94254d8ca45fbe7186863da0d4ab08b"
server_key_iv: "b4b37e2439b1d514a650a4f894d48e91"
server_key_version: "encryptionkey@0"
I20260927 04:59:41.546411 27733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:41.567369 27733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:41.571694 27733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:41.571987 27733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:41.572947 27733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:41.574286 27733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:41.574532 27733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:41.574705 27733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:41.574779 27733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:41.611661 27733 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:41045
I20260927 04:59:41.611703 27861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:41045 every 8 connection(s)
I20260927 04:59:41.613020 27733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:41.617153 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27733
I20260927 04:59:41.617347 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:41.617811 21157 external_mini_cluster.cc:1428] Setting key 83687ef2e06fd1cd324249f027609aa1
Sep 27 04:59:41 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485181, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:41.628021 27593 heartbeater.cc:503] Master 127.20.169.126:36927 was elected leader, sending a full tablet report...
I20260927 04:59:41.633759 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:41.615435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:55913 (local address 127.20.169.126:36927)
0927 04:59:41.615616 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:41.615622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:41.616631 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 04:59:41.618664 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:41.618669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:41.618736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:41.618851 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:41.620534 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.621405 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.622905 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.623307 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.623450 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:41.626992 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:41.626998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:41.627001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:41.627034 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:41.630378 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:41.631156 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:41.631164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:41.631168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:41.631340 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:41.631769 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:41.631775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:41.631779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:41.632193 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:41.632370 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:41.633319 (+ 949us) server_negotiation.cc:299] Negotiation successful
0927 04:59:41.633504 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 04:59:41.634756 27864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:41.615765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:55913)
0927 04:59:41.616466 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:41.616508 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:41.618344 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:41.619107 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:41.619119 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:41.619705 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:41.620374 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:41.620391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.621540 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.621544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:41.622751 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:41.622762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:41.623552 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:41.623558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:41.623787 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:41.624487 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:41.624515 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:41.626719 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:41.630522 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:41.630532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:41.630548 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:41.631002 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:41.631467 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:41.631471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:41.631473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:41.631641 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:41.632352 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:41.632357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:41.632782 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 04:59:41.633230 (+ 448us) client_negotiation.cc:240] Negotiation successful
0927 04:59:41.633647 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260927 04:59:41.636067 27862 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36927
I20260927 04:59:41.636350 27862 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:41.637166 27862 heartbeater.cc:511] Master 127.20.169.126:36927 requested a full tablet report, sending...
I20260927 04:59:41.638700 27403 ts_manager.cc:194] Registered new tserver with Master: 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045)
I20260927 04:59:41.639559 27403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:55913
I20260927 04:59:41.645290 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.169.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.125.keytab.
Entry for principal oryx/127.20.169.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.125.keytab.
I20260927 04:59:41.696063 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:45003
--webserver_interface=127.20.169.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.125.keytab
--principal=oryx/127.20.169.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:36927,127.20.169.125:45003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:42.027920 27867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:42.028193 27867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:42.028389 27867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:42.035766 27867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:42.035874 27867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:42.035961 27867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:42.036062 27867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:42.036129 27867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:42.046752 27867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:36927,127.20.169.125:45003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.125.keytab
--principal=oryx/127.20.169.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.169.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
W20260927 04:59:42.047152 27867 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.169.126:36927,127.20.169.125:45003'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 04:59:42.048163 27867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:42.049520 27867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:42.061327 27873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:42.061327 27872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:42.061913 27875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:42.062166 27867 server_base.cc:1034] running on GCE node
I20260927 04:59:42.063019 27867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:42.064409 27867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:42.065645 27867 hybrid_clock.cc:648] HybridClock initialized: now 1790485182065583 us; error 39 us; skew 500 ppm
I20260927 04:59:42.065994 27867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:42.073506 27867 init.cc:377] Logged in from keytab as oryx/127.20.169.125@KRBTEST.COM (short username oryx)
I20260927 04:59:42.074909 27867 webserver.cc:460] Webserver started at http://127.20.169.125:43741/ using document root <none> and password file <none>
I20260927 04:59:42.075591 27867 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:42.075716 27867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:42.076015 27867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:42.078896 27867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/instance:
uuid: "3543ce954bc14f5f898b6106cbd69262"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "33eab9e8454c4fd5000366b63bb016bb"
server_key_iv: "da1d4c7cc6823f1eadda85520a36c9f5"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.079636 27867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal/instance:
uuid: "3543ce954bc14f5f898b6106cbd69262"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "33eab9e8454c4fd5000366b63bb016bb"
server_key_iv: "da1d4c7cc6823f1eadda85520a36c9f5"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.084630 27867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 04:59:42.087926 27882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:42.089327 27867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:42.089502 27867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
uuid: "3543ce954bc14f5f898b6106cbd69262"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "33eab9e8454c4fd5000366b63bb016bb"
server_key_iv: "da1d4c7cc6823f1eadda85520a36c9f5"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.089715 27867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:42.105309 27867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:42.109680 27867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:42.109947 27867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:42.128054 27867 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.125:45003
I20260927 04:59:42.128073 27934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.125:45003 every 8 connection(s)
I20260927 04:59:42.129441 27867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
I20260927 04:59:42.130566 27728 heartbeater.cc:503] Master 127.20.169.126:36927 was elected leader, sending a full tablet report...
I20260927 04:59:42.134020 27935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:42.138429 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27867
I20260927 04:59:42.141968 27935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.125@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:42.160853 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.146053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.20.169.126:36927)
0927 04:59:42.146522 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.146531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.147051 (+ 520us) server_negotiation.cc:407] Connection header received
0927 04:59:42.147258 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.147264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.147382 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.147514 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.148589 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.149365 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.150880 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.151364 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.151496 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.155078 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.155083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.155086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.155120 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.157842 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.158547 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.158554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.158557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.158689 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.159086 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.159091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.159094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.159416 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.159581 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.160122 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.160419 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
I20260927 04:59:42.161866 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.140586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.20.169.125:45003)
0927 04:59:42.141532 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.141544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.141582 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:59:42.143251 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.143272 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.143703 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.144065 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.145885 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.147965 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.149252 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.150066 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.150251 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.154746 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.154754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.154767 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.154798 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.157754 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.158474 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.158481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.158483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.158627 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.159022 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.159031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.159033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.159527 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.159835 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.160174 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.160578 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":747,"thread_start_us":446,"threads_started":1}
I20260927 04:59:42.161882 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.146035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:59036)
0927 04:59:42.146852 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.146900 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.147120 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.147581 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.147587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.148204 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.148445 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.148453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.149521 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.149525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.150730 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.150741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.151591 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.151597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.151746 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.152392 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.152414 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.154880 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.157980 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.157991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.158004 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.158436 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.158807 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.158812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.158816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.158973 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.159555 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.159564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.159766 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.160044 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.160722 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":153,"thread_start_us":476,"threads_started":1}
I20260927 04:59:42.164496 27935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.125@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
I20260927 04:59:42.177073 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.165248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:42612)
0927 04:59:42.165515 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.165540 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.165618 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.165870 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.165874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.166170 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.166299 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.166307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.167224 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.167227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.168072 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.168087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.168741 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.168745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.168955 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.169485 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.169501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.171837 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.174810 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.174814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.174817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.175196 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.175509 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.175513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.175515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.175644 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.176183 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.176187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.176387 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.176512 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.176731 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 04:59:42.177304 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.165438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.20.169.125:45003)
0927 04:59:42.165586 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.165590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.165604 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:59:42.165708 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.165711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.165762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.165842 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.166423 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.167099 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.168212 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.168614 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.168713 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.172019 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.172024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.172028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.172055 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.174676 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.175305 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.175308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.175310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.175415 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.175747 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.175751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.175753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.176066 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.176171 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.176602 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.176968 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":54}
I20260927 04:59:42.182152 27935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Bootstrap starting.
I20260927 04:59:42.185331 27935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:42.186354 27935 log.cc:824] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:42.189034 27935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: No bootstrap required, opened a new log
I20260927 04:59:42.195751 27935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } }
I20260927 04:59:42.196081 27935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:42.196185 27935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3543ce954bc14f5f898b6106cbd69262, State: Initialized, Role: FOLLOWER
I20260927 04:59:42.196785 27935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } }
I20260927 04:59:42.197705 27942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } } }
I20260927 04:59:42.198072 27942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:59:42.198680 27935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 04:59:42.207650 27953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:59:42.207785 27953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:59:42.210511 27867 master_runner.cc:428] Detected that this master 3543ce954bc14f5f898b6106cbd69262 is joining an existing cluster
I20260927 04:59:42.210628 27867 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.169.125:45003
I20260927 04:59:42.230908 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.214480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.20.169.126:36927)
0927 04:59:42.214668 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.214676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.215147 (+ 471us) server_negotiation.cc:407] Connection header received
0927 04:59:42.215440 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.215447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.215544 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.215621 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.217123 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.218248 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.220493 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.221228 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.221356 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.224056 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.224064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.224071 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.224115 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.227735 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.228532 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.228538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.228542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.228694 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.229190 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.229195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.229198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.229558 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.229680 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.230319 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.230528 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 04:59:42.231403 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.214429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:59040)
0927 04:59:42.215019 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.215060 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.215258 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.215836 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.215841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.216503 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.216946 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.216961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.218447 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.218455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.220328 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.220345 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.221658 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.221666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.221892 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.222607 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.222635 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.223852 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.227875 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.227880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.227884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.228385 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.228875 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.228880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.228884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.229046 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.229788 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.229799 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.229999 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.230231 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.230844 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":322,"threads_started":1}
I20260927 04:59:42.243098 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.215269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:42628)
0927 04:59:42.225251 (+ 9982us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.225267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.225452 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.225930 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.225937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.226575 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.226905 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.226918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.228387 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.228391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.230000 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.230010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.231722 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.231733 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.231975 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.232575 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.232605 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.233938 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.238244 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.238249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.238253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.238776 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.239897 (+ 1121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.239905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.239911 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.240104 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.241607 (+ 1503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.241614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.241794 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.242073 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.242460 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9842,"thread_start_us":9767,"threads_started":1}
I20260927 04:59:42.244632 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.215538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.20.169.125:45003)
0927 04:59:42.215853 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.215859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.225408 (+ 9549us) server_negotiation.cc:407] Connection header received
0927 04:59:42.225586 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.225593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.225678 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.225769 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.227054 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.228179 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.230166 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.230992 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.231129 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.234142 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.234150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.234156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.234192 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.238055 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.239442 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.239454 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.239460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.239638 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.240242 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.240249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.240255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.240737 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.241550 (+ 813us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.243841 (+ 2291us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.244123 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
WARNING: no policy specified for oryx/127.20.169.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.124@KRBTEST.COM" created.
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.126@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
I20260927 04:59:42.265601 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.239030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.20.169.125:45003)
0927 04:59:42.246437 (+ 7407us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.246442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.249954 (+ 3512us) server_negotiation.cc:407] Connection header received
0927 04:59:42.250509 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.250513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.250598 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.250718 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.253659 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.254425 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.256127 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.256616 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.256755 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.259948 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.259953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.259956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.259984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.262754 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.263459 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.263467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.263471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.263610 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.264027 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.264030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.264032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.264281 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.264497 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.264995 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.265203 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7305,"thread_start_us":9962,"threads_started":1}
I20260927 04:59:42.265642 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.238666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:42632)
0927 04:59:42.249790 (+ 11124us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.249851 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.250098 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.250775 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.250784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.253268 (+ 2484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.253482 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.253496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.254559 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.254564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.255979 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.255991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.256835 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.256845 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.257027 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.257695 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.257712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.259767 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.262905 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.262918 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.262925 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.263315 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.263721 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.263728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.263733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.263892 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.264442 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.264468 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.264658 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.264860 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.265200 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10830,"thread_start_us":10619,"threads_started":1}
I20260927 04:59:42.267707 27403 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.169.125:45003: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3543ce954bc14f5f898b6106cbd69262"
member_type: NON_VOTER
last_known_addr {
host: "127.20.169.125"
port: 45003
}
attrs {
promote: true
}
}
dest_uuid: "99fa63b96cd341f9b51f24e007bff59d"
cas_config_opid_index: -1
I20260927 04:59:42.268667 27403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } }
I20260927 04:59:42.283802 27966 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3543ce954bc14f5f898b6106cbd69262 (127.20.169.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } }
I20260927 04:59:42.285089 27909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:42.287802 27966 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.169.125:45003
I20260927 04:59:42.288596 27867 master.cc:537] Master@127.20.169.125:45003 shutting down...
I20260927 04:59:42.288272 27967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } }
I20260927 04:59:42.289721 27967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:42.290221 27909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Refusing update from remote peer 99fa63b96cd341f9b51f24e007bff59d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 04:59:42.291519 27966 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.124.keytab.
Entry for principal oryx/127.20.169.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.124.keytab.
I20260927 04:59:42.294660 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:45259
--webserver_interface=127.20.169.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.124.keytab
--principal=oryx/127.20.169.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:42.295910 27387 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 04:59:42.300083 27387 consensus_peers.cc:487] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d -> Peer 3543ce954bc14f5f898b6106cbd69262 (127.20.169.125:45003): Couldn't send request to peer 3543ce954bc14f5f898b6106cbd69262. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 04:59:42.302100 27867 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:42.302433 27867 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:42.302567 27867 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: stopping tablet replica
I20260927 04:59:42.309577 27867 master.cc:552] Master@127.20.169.125:45003 shutdown complete.
I20260927 04:59:42.309684 27867 master_runner.cc:305] Clearing existing system tablet
I20260927 04:59:42.310093 27867 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:42.310190 27867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:42.312971 27867 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 04:59:42.314541 27972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:42.314919 27867 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:42.315214 27867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 04:59:42.315336 27867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
uuid: "3543ce954bc14f5f898b6106cbd69262"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "33eab9e8454c4fd5000366b63bb016bb"
server_key_iv: "da1d4c7cc6823f1eadda85520a36c9f5"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.318812 27867 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:42.323271 27867 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 04:59:42.323465 27867 log.cc:1196] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 04:59:42.323890 27867 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Deleting consensus metadata
I20260927 04:59:42.324147 27867 master_runner.cc:315] Copying system tablet from 127.20.169.126:36927
I20260927 04:59:42.330119 27867 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: tablet copy: Beginning tablet copy session from remote peer at address 127.20.169.126:36927
I20260927 04:59:42.344765 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.330958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.20.169.126:36927)
0927 04:59:42.331198 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.331208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.333446 (+ 2238us) server_negotiation.cc:407] Connection header received
0927 04:59:42.333662 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.333668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.333738 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.333851 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.334680 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.335435 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.336626 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.337220 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.337360 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.339291 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.339298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.339303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.339340 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.342220 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.342818 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.342823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.342827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.342961 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.343348 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.343354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.343357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.343628 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.343785 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.344164 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.344372 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 04:59:42.344853 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.330952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:59046)
0927 04:59:42.333327 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.333347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.333520 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.333884 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.333888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.334310 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.334532 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.334541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.335557 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.335563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.336485 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.336496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.337409 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.337416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.337615 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.338240 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.338258 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.339005 (+ 747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.342338 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.342344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.342348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.342694 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.343068 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.343073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.343077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.343234 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.343750 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.343754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.343929 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.344070 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.344382 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"thread_start_us":2124,"threads_started":1}
I20260927 04:59:42.346128 27423 tablet_copy_service.cc:140] P 99fa63b96cd341f9b51f24e007bff59d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3543ce954bc14f5f898b6106cbd69262 ({username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:59046)
I20260927 04:59:42.346417 27423 tablet_copy_service.cc:161] P 99fa63b96cd341f9b51f24e007bff59d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3543ce954bc14f5f898b6106cbd69262 at {username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:59046: session id = 3543ce954bc14f5f898b6106cbd69262-00000000000000000000000000000000
I20260927 04:59:42.349903 27423 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 04:59:42.352123 27867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 04:59:42.356777 27867 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: tablet copy: Starting download of 0 data blocks...
I20260927 04:59:42.357137 27867 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: tablet copy: Starting download of 1 WAL segments...
I20260927 04:59:42.359575 27867 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 04:59:42.362792 27423 tablet_copy_service.cc:342] P 99fa63b96cd341f9b51f24e007bff59d: Request end of tablet copy session 3543ce954bc14f5f898b6106cbd69262-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:59046
I20260927 04:59:42.363013 27423 tablet_copy_service.cc:433] P 99fa63b96cd341f9b51f24e007bff59d: ending tablet copy session 3543ce954bc14f5f898b6106cbd69262-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3543ce954bc14f5f898b6106cbd69262
I20260927 04:59:42.373320 27867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:42.376082 27979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:42.378206 27980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:42.389544 27982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:42.394061 27867 server_base.cc:1034] running on GCE node
I20260927 04:59:42.394451 27867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:42.395126 27867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:42.396433 27867 hybrid_clock.cc:648] HybridClock initialized: now 1790485182396358 us; error 37 us; skew 500 ppm
I20260927 04:59:42.396657 27867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:42.398535 27867 init.cc:377] Logged in from keytab as oryx/127.20.169.125@KRBTEST.COM (short username oryx)
I20260927 04:59:42.401618 27867 webserver.cc:460] Webserver started at http://127.20.169.125:37357/ using document root <none> and password file <none>
I20260927 04:59:42.401948 27867 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:42.402040 27867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:42.404593 27867 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 04:59:42.406442 27989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:42.406924 27867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:59:42.407078 27867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
uuid: "3543ce954bc14f5f898b6106cbd69262"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "33eab9e8454c4fd5000366b63bb016bb"
server_key_iv: "da1d4c7cc6823f1eadda85520a36c9f5"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.407241 27867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:42.428246 27867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:42.431512 27867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:42.464946 27867 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.125:45003
I20260927 04:59:42.464986 28040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.125:45003 every 8 connection(s)
I20260927 04:59:42.465900 27867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
I20260927 04:59:42.473017 28041 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:59:42.475459 28041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Bootstrap starting.
I20260927 04:59:42.488170 28041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:59:42.488686 28041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Bootstrap complete.
I20260927 04:59:42.489566 28041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } }
I20260927 04:59:42.489807 28041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3543ce954bc14f5f898b6106cbd69262, State: Initialized, Role: LEARNER
I20260927 04:59:42.490070 28041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } }
I20260927 04:59:42.490820 28044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } }
I20260927 04:59:42.491154 28041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:42.491178 28044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:59:42.499260 28055 catalog_manager.cc:1259] Loaded cluster ID: 56befa11045343c0be45d783eb3564eb
I20260927 04:59:42.499372 28055 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: loading cluster ID for follower catalog manager: success
I20260927 04:59:42.502398 28055 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: acquiring CA information for follower catalog manager: success
I20260927 04:59:42.504334 28055 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:59:42.642372 27862 heartbeater.cc:503] Master 127.20.169.126:36927 was elected leader, sending a full tablet report...
W20260927 04:59:42.815299 27970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:42.816092 27970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:42.816453 27970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 04:59:42.828727 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.815424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:60592)
0927 04:59:42.816273 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.816293 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.816496 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.818102 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.818109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.818180 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:42.818570 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.818593 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.821774 (+ 3181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.821781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.824843 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.824863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.827668 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.827674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.827923 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.827988 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.828127 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.828268 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":562,"threads_started":1}
I20260927 04:59:42.829012 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.816011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.20.169.125:45003)
0927 04:59:42.817595 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.817602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.817642 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:59:42.817713 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.817719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.817813 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.817949 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:42.818779 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.821455 (+ 2676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.825043 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.827357 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.827819 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.828007 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.828164 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.828340 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":1004,"threads_started":1}
W20260927 04:59:42.830972 27970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:42.831223 27970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:42.831405 27970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:42.831610 27970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:42.831769 27970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:42.831715 28016 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 04:59:42.833062 28044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99fa63b96cd341f9b51f24e007bff59d. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } }
I20260927 04:59:42.833451 28044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:59:42.846177 27970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.124.keytab
--principal=oryx/127.20.169.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.169.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:42.847842 27970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:42.849359 27970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:42.867146 28063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:42.869498 28066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:42.869846 28064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:42.870699 27970 server_base.cc:1034] running on GCE node
I20260927 04:59:42.871803 27970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:42.874130 27970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:42.875414 27970 hybrid_clock.cc:648] HybridClock initialized: now 1790485182875273 us; error 93 us; skew 500 ppm
I20260927 04:59:42.875820 27970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:42.880283 27970 init.cc:377] Logged in from keytab as oryx/127.20.169.124@KRBTEST.COM (short username oryx)
I20260927 04:59:42.882237 27970 webserver.cc:460] Webserver started at http://127.20.169.124:38721/ using document root <none> and password file <none>
I20260927 04:59:42.883104 27970 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:42.883256 27970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:42.883636 27970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:42.887295 27970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/instance:
uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "f1bce65b1ebea47ee38bde026ea878d2"
server_key_iv: "68a94bee2f4c284354c162ffd0b74384"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.888239 27970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal/instance:
uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "f1bce65b1ebea47ee38bde026ea878d2"
server_key_iv: "68a94bee2f4c284354c162ffd0b74384"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.895290 27970 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260927 04:59:42.899955 28073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:42.901465 27970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 04:59:42.901649 27970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "f1bce65b1ebea47ee38bde026ea878d2"
server_key_iv: "68a94bee2f4c284354c162ffd0b74384"
server_key_version: "encryptionkey@0"
I20260927 04:59:42.901867 27970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:42.922430 27970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:42.928064 27970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:42.928532 27970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:42.953510 27970 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.124:45259
I20260927 04:59:42.953558 28125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.124:45259 every 8 connection(s)
I20260927 04:59:42.954921 27970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
I20260927 04:59:42.956537 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 27970
I20260927 04:59:42.959944 28126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:42.967185 28126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.124@KRBTEST.COM
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.124@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:42.981014 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.958923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.20.169.124:45259)
0927 04:59:42.959696 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.959709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.959751 (+ 42us) server_negotiation.cc:407] Connection header received
0927 04:59:42.961511 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.961534 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.961916 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.962300 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.963770 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.965190 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.966414 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.967064 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.967255 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.972755 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.972767 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.972780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.972877 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.976305 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.977336 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.977348 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.977355 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.977639 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.978034 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.978040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.978043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.978742 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.978949 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.979596 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.979996 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"spinlock_wait_cycles":28416,"thread_start_us":361,"threads_started":1}
I20260927 04:59:42.986809 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.970151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:48874)
0927 04:59:42.970959 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.970999 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.971132 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.971799 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.971805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.972396 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.972698 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.972715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.973910 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.973941 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.975712 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.975726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.976553 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.976558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.976671 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.977359 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.977385 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.980203 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.984404 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.984412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.984418 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.984932 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.985296 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.985299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.985302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:42.985469 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:42.985964 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:42.985969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:42.986106 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 04:59:42.986265 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 04:59:42.986517 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":447,"threads_started":1}
I20260927 04:59:42.986809 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.970289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.20.169.126:36927)
0927 04:59:42.971492 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.971497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.971511 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:59:42.971556 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.971560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.971615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.971730 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.972915 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.973753 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.975877 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.976326 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.976532 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.980411 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.980422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.980426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.980459 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.984251 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.985059 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.985063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.985065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.985196 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:42.985574 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:42.985578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:42.985580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:42.985847 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:42.985957 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:42.986370 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 04:59:42.986530 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":1021,"threads_started":1}
I20260927 04:59:42.988137 28126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:59:42 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.124@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
I20260927 04:59:43.002573 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.988863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:60596)
0927 04:59:42.989073 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:42.989096 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:42.989199 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:42.989557 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:42.989562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:42.989921 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:42.990075 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.990084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.991186 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.991192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.992191 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:42.992202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.993019 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.993025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:42.993226 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.993782 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:42.993800 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:42.995988 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:42.999519 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:42.999525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:42.999529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.000051 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.000518 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.000523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.000527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.000723 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.001354 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.001359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.001529 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.001693 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.002065 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 04:59:43.002592 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:42.989024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.20.169.125:45003)
0927 04:59:42.989237 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:42.989247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:42.989280 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:59:42.989349 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:42.989355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:42.989428 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:42.989540 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:42.990230 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.991038 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.992350 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:42.992863 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:42.993010 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:42.996202 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:42.996207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:42.996210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:42.996248 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:42.999370 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.000191 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.000205 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.000207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.000384 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.000864 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.000872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.000876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.001223 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.001391 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.001794 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.002076 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 04:59:43.003717 28126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485182, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.124@KRBTEST.COM for oryx/127.20.169.124@KRBTEST.COM
I20260927 04:59:43.020553 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.004349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38282)
0927 04:59:43.004733 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.004761 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.004979 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.005473 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.005476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.005996 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.006261 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.006270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.007779 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.007787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.008994 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.009005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.009855 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.009860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.010189 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.010785 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.010800 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.013328 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.017243 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.017250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.017255 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.017800 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.018306 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.018313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.018320 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.018546 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.019157 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.019162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.019335 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.019509 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.019827 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 04:59:43.020640 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.004958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.20.169.124:45259)
0927 04:59:43.005217 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.005223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.005240 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:43.005284 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.005289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.005350 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.005440 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.006436 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.007667 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.009118 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.009680 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.009828 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.013537 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.013541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.013544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.013570 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.017108 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.017932 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.017947 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.017951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.018153 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.018672 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.018679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.018688 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.019024 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.019181 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.019638 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.020060 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":150}
I20260927 04:59:43.026103 28126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Bootstrap starting.
I20260927 04:59:43.030269 28126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:43.031718 28126 log.cc:824] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:43.035192 28126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: No bootstrap required, opened a new log
I20260927 04:59:43.045547 28126 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } }
I20260927 04:59:43.045992 28126 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:43.046202 28126 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c4aae6ac914b81b1ce0d683ef12183, State: Initialized, Role: FOLLOWER
I20260927 04:59:43.046823 28126 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } }
I20260927 04:59:43.048112 28133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } } }
I20260927 04:59:43.048843 28133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:59:43.049243 28126 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 04:59:43.064918 28144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:59:43.065119 28144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:59:43.070465 27970 master_runner.cc:428] Detected that this master f6c4aae6ac914b81b1ce0d683ef12183 is joining an existing cluster
I20260927 04:59:43.070652 27970 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.169.124:45259
I20260927 04:59:43.087472 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27464
I20260927 04:59:43.109148 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.077727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.20.169.126:36927)
0927 04:59:43.077957 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.077965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.078754 (+ 789us) server_negotiation.cc:407] Connection header received
0927 04:59:43.079028 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.079034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.079116 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.079191 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.081448 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.082597 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.084957 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.085632 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.085741 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.096448 (+ 10707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.096455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.096460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.096503 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.101558 (+ 5055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.102606 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.102611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.102618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.102876 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.103517 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.103531 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.103535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.104073 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.104274 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.108076 (+ 3802us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.108411 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.65@KRBTEST.COM
I20260927 04:59:43.109640 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.078980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.20.169.125:45003)
0927 04:59:43.079216 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.079225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.080524 (+ 1299us) server_negotiation.cc:407] Connection header received
0927 04:59:43.080828 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.080836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.080947 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.081032 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.082840 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.084274 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.086972 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.087644 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.087739 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.092969 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.092976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.092981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.093024 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.103828 (+ 10804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.106237 (+ 2409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.106243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.106256 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.106497 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.107064 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.107070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.107075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.107571 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.107718 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.108780 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.109150 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 04:59:43.111306 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.078845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:60610)
0927 04:59:43.080417 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.080438 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.080636 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.081292 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.081299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.082215 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.082613 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.082628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.084507 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.084515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.086767 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.086798 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.088330 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.088336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.088565 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.089945 (+ 1380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.089976 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.092707 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.105448 (+ 12741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.105464 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.105468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.106067 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.106658 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.106662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.106666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.106900 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.107795 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.107799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.108428 (+ 629us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.109852 (+ 1424us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.110280 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"thread_start_us":1028,"threads_started":1}
I20260927 04:59:43.115056 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.077924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:48888)
0927 04:59:43.078654 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.078674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.078859 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.079410 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.079414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.080668 (+ 1254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.081202 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.081218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.082892 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.082898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.084741 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.084757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.089632 (+ 4875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.089650 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.091346 (+ 1696us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.093575 (+ 2229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.093618 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.096077 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.101813 (+ 5736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.101818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.101822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.102434 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.103057 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.103061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.103066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.103351 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.104399 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.104403 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.107624 (+ 3221us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.110846 (+ 3222us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.111347 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":696,"thread_start_us":421,"threads_started":1}
I20260927 04:59:43.125214 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.088910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.20.169.65:41387)
0927 04:59:43.090979 (+ 2069us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.090990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.091021 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:59:43.091120 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.091128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.091411 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.095599 (+ 4188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.095985 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.098261 (+ 2276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.101064 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.102403 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.111806 (+ 9403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.112286 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.112300 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.112321 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.112371 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.118482 (+ 6111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.119365 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.119378 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.119382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.119640 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.120511 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.120517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.120522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.121312 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.123684 (+ 2372us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.123938 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.124337 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1830,"thread_start_us":1510,"threads_started":1}
I20260927 04:59:43.126735 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.080345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38294)
0927 04:59:43.087023 (+ 6678us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.087043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.087150 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.088120 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.088125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.091245 (+ 3120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.092382 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.092397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.095820 (+ 3423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.095829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.106301 (+ 10472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.106326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.110080 (+ 3754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.110087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.110441 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.111736 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.111777 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.113951 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.122247 (+ 8296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.122252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.122266 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.123247 (+ 981us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.123793 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.123798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.123812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.124045 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.125090 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.125095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.125335 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.125599 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.125962 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6528,"mutex_wait_us":22,"thread_start_us":6859,"threads_started":1}
I20260927 04:59:43.127251 27588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:43.128113 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.077483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.20.169.124:45259)
0927 04:59:43.077673 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.077679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.087895 (+ 10216us) server_negotiation.cc:407] Connection header received
0927 04:59:43.087936 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.087939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.087990 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.088103 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.092546 (+ 4443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.094183 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.106485 (+ 12302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.107779 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.107944 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.114300 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.114368 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.114374 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.114411 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.122056 (+ 7645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.123412 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.123417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.123421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.123646 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.124187 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.124192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.124196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.124835 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.125015 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.127269 (+ 2254us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.127599 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.126@KRBTEST.COM for oryx/127.20.169.124@KRBTEST.COM
I20260927 04:59:43.152051 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.123202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38300)
0927 04:59:43.123428 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.123447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.123555 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.131208 (+ 7653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.131214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.131962 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.132294 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.132404 (+ 110us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.135193 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.135201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.137498 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.137520 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.139281 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.139289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.139584 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.140652 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.140680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.144604 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.148745 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.148750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.148754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.149366 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.149834 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.149839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.149843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.150042 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.150836 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.150844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.151069 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.151205 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.151521 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 04:59:43.152120 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.123636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.20.169.124:45259)
0927 04:59:43.130687 (+ 7051us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.130693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.130712 (+ 19us) server_negotiation.cc:407] Connection header received
0927 04:59:43.130845 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.130850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.130990 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.132656 (+ 1666us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.132996 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.134844 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.137816 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.139018 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.140113 (+ 1095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.144847 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.144857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.144862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.144896 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.148609 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.149551 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.149558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.149561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.149729 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.150168 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.150173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.150176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.150669 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.150823 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.151301 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.151561 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6929,"thread_start_us":11695,"threads_started":1}
I20260927 04:59:43.154232 27403 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.169.124:45259: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
member_type: NON_VOTER
last_known_addr {
host: "127.20.169.124"
port: 45259
}
attrs {
promote: true
}
}
dest_uuid: "99fa63b96cd341f9b51f24e007bff59d"
cas_config_opid_index: 5
I20260927 04:59:43.155396 27403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } }
I20260927 04:59:43.157536 28059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.162024 28016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 LEARNER]: Refusing update from remote peer 99fa63b96cd341f9b51f24e007bff59d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 04:59:43.164701 28100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:43.165789 28156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 04:59:43.174736 28059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.169.124:45259
I20260927 04:59:43.176326 28100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Refusing update from remote peer 99fa63b96cd341f9b51f24e007bff59d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 04:59:43.177378 27970 master.cc:537] Master@127.20.169.124:45259 shutting down...
I20260927 04:59:43.177687 28156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.179194 28156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:43.179968 28059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 04:59:43.181700 28079 tablet.cc:2369] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:43.193176 27970 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:43.193833 27970 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:43.194173 27970 tablet_replica.cc:330] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: stopping tablet replica
I20260927 04:59:43.193907 28016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
W20260927 04:59:43.202988 27388 consensus_peers.cc:487] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d -> Peer f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124:45259): Couldn't send request to peer f6c4aae6ac914b81b1ce0d683ef12183. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:59:43.203024 27970 master.cc:552] Master@127.20.169.124:45259 shutdown complete.
I20260927 04:59:43.203121 27970 master_runner.cc:305] Clearing existing system tablet
I20260927 04:59:43.203528 27970 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:43.203639 27970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:43.206063 28156 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: attempting to promote NON_VOTER 3543ce954bc14f5f898b6106cbd69262 to VOTER
I20260927 04:59:43.206024 28044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: NON_VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: true } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.206301 28044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:59:43.207288 28156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } }
I20260927 04:59:43.209838 28016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 LEARNER]: Refusing update from remote peer 99fa63b96cd341f9b51f24e007bff59d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 04:59:43.210098 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.209520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38314)
0927 04:59:43.209666 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.209828 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:59:43.211323 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.210906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38320)
0927 04:59:43.211064 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.211139 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:59:43.211246 28156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 04:59:43.211899 27388 consensus_peers.cc:487] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d -> Peer f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124:45259): Couldn't send request to peer f6c4aae6ac914b81b1ce0d683ef12183. Status: Network error: Client connection negotiation failed: client connection to 127.20.169.124:45259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:59:43.214332 28157 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3543ce954bc14f5f898b6106cbd69262 (127.20.169.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.215423 28016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3543ce954bc14f5f898b6106cbd69262 (127.20.169.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.220050 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.220579 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:43.221065 28156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.221469 28156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:43.222797 28044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.223114 28044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:59:43.223779 27970 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.005s sys 0.000s
I20260927 04:59:43.226804 28162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:43.227439 27970 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:43.227795 27970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 04:59:43.227965 27970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "f1bce65b1ebea47ee38bde026ea878d2"
server_key_iv: "68a94bee2f4c284354c162ffd0b74384"
server_key_version: "encryptionkey@0"
I20260927 04:59:43.233891 27970 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:43.243046 27970 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 04:59:43.243273 27970 log.cc:1196] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 04:59:43.243954 27970 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Deleting consensus metadata
I20260927 04:59:43.244247 27970 master_runner.cc:315] Copying system tablet from 127.20.169.126:36927
I20260927 04:59:43.248387 27970 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: tablet copy: Beginning tablet copy session from remote peer at address 127.20.169.126:36927
I20260927 04:59:43.269754 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.249341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.20.169.126:36927)
0927 04:59:43.249775 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.249793 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.250533 (+ 740us) server_negotiation.cc:407] Connection header received
0927 04:59:43.250896 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.250902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.250983 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.251059 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.252672 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.254062 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.256291 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.257186 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.257325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.261005 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.261012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.261017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.261056 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.265826 (+ 4770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.266897 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.266903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.266907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.267144 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.267665 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.267670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.267674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.268107 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.268267 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.268935 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.269160 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 04:59:43.270232 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.249341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:48892)
0927 04:59:43.250424 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.250443 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.250672 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.251286 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.251290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.252099 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.252503 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.252515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.254293 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.254298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.256126 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.256145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.257507 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.257515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.257885 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.258961 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.258992 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.260667 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.266040 (+ 5373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.266045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.266048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.266720 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.267278 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.267283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.267287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.267531 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.268337 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.268350 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.268582 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.268753 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.269217 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":709,"threads_started":1}
I20260927 04:59:43.271044 27423 tablet_copy_service.cc:140] P 99fa63b96cd341f9b51f24e007bff59d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f6c4aae6ac914b81b1ce0d683ef12183 ({username='oryx', principal='oryx/127.20.169.124@KRBTEST.COM'} at 127.0.0.1:48892)
I20260927 04:59:43.271312 27423 tablet_copy_service.cc:161] P 99fa63b96cd341f9b51f24e007bff59d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f6c4aae6ac914b81b1ce0d683ef12183 at {username='oryx', principal='oryx/127.20.169.124@KRBTEST.COM'} at 127.0.0.1:48892: session id = f6c4aae6ac914b81b1ce0d683ef12183-00000000000000000000000000000000
I20260927 04:59:43.275794 27423 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 04:59:43.277968 27970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 04:59:43.287876 27970 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: tablet copy: Starting download of 0 data blocks...
I20260927 04:59:43.288300 27970 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: tablet copy: Starting download of 1 WAL segments...
I20260927 04:59:43.291651 27970 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 04:59:43.297616 27423 tablet_copy_service.cc:342] P 99fa63b96cd341f9b51f24e007bff59d: Request end of tablet copy session f6c4aae6ac914b81b1ce0d683ef12183-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.169.124@KRBTEST.COM'} at 127.0.0.1:48892
I20260927 04:59:43.297802 27423 tablet_copy_service.cc:433] P 99fa63b96cd341f9b51f24e007bff59d: ending tablet copy session f6c4aae6ac914b81b1ce0d683ef12183-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f6c4aae6ac914b81b1ce0d683ef12183
I20260927 04:59:43.302004 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27464
I20260927 04:59:43.310799 27970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:43.313746 28169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:43.318318 28170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:43.328902 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:41387
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:36927
--webserver_port=44817
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:43.330384 28172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:43.331091 27970 server_base.cc:1034] running on GCE node
I20260927 04:59:43.331504 27970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:43.332283 27970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:43.333521 27970 hybrid_clock.cc:648] HybridClock initialized: now 1790485183333478 us; error 37 us; skew 500 ppm
I20260927 04:59:43.333773 27970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485183, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:43.335925 27970 init.cc:377] Logged in from keytab as oryx/127.20.169.124@KRBTEST.COM (short username oryx)
I20260927 04:59:43.343250 27970 webserver.cc:460] Webserver started at http://127.20.169.124:44573/ using document root <none> and password file <none>
I20260927 04:59:43.343600 27970 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:43.343698 27970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:43.347096 27970 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 04:59:43.349116 28180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:43.349578 27970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 04:59:43.349740 27970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
format_stamp: "Formatted at 2026-09-27 04:59:42 on dist-test-slave-39f2"
server_key: "f1bce65b1ebea47ee38bde026ea878d2"
server_key_iv: "68a94bee2f4c284354c162ffd0b74384"
server_key_version: "encryptionkey@0"
I20260927 04:59:43.349999 27970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:43.367053 27970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:43.371372 27970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:43.405323 27970 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.124:45259
I20260927 04:59:43.405416 28231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.124:45259 every 8 connection(s)
I20260927 04:59:43.406427 27970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
I20260927 04:59:43.412248 28232 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:59:43.416389 28232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Bootstrap starting.
I20260927 04:59:43.436331 28232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:59:43.436937 28232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: Bootstrap complete.
I20260927 04:59:43.438129 28232 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } }
I20260927 04:59:43.438446 28232 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6c4aae6ac914b81b1ce0d683ef12183, State: Initialized, Role: LEARNER
I20260927 04:59:43.438750 28232 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } }
I20260927 04:59:43.439540 28235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.439971 28235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:59:43.440038 28232 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:43.453895 28246 catalog_manager.cc:1259] Loaded cluster ID: 56befa11045343c0be45d783eb3564eb
I20260927 04:59:43.453992 28246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: loading cluster ID for follower catalog manager: success
I20260927 04:59:43.457201 28246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: acquiring CA information for follower catalog manager: success
I20260927 04:59:43.459429 28246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 04:59:43.708992 28175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:43.709367 28175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:43.709592 28175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:43.718724 28175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:43.718936 28175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:43.719158 28175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:43.729720 28175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:41387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=44817
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:43.731048 28175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:43.732421 28175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:43.747087 28251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:43.747141 28254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:43.747165 28252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:43.748194 28175 server_base.cc:1034] running on GCE node
I20260927 04:59:43.748769 28175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:43.749727 28175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:43.750967 28175 hybrid_clock.cc:648] HybridClock initialized: now 1790485183750912 us; error 41 us; skew 500 ppm
I20260927 04:59:43.751385 28175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485183, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:43.754793 28175 init.cc:377] Logged in from keytab as oryx/127.20.169.65@KRBTEST.COM (short username oryx)
I20260927 04:59:43.756278 28175 webserver.cc:460] Webserver started at http://127.20.169.65:44817/ using document root <none> and password file <none>
I20260927 04:59:43.757088 28175 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:43.757251 28175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:43.763172 28175 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 04:59:43.766202 28261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:43.767647 28175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:43.767866 28175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "f245a22d06014d6d9f47477d25dcd6fc"
format_stamp: "Formatted at 2026-09-27 04:59:40 on dist-test-slave-39f2"
server_key: "f22461234dd644742b2144fb61ae651d"
server_key_iv: "16aa6bc582e4884ebb266646747c806b"
server_key_version: "encryptionkey@0"
I20260927 04:59:43.768965 28175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:43.785331 28175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:43.791770 28175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:43.792232 28175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:43.793890 28175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:43.795749 28175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:43.795955 28175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:43.796119 28175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:43.796311 28175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:43.814215 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.803427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38322)
0927 04:59:43.804320 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.804348 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.804495 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.805402 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.805407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.805435 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:43.805853 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.805870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.809302 (+ 3432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.809309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.810808 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.810841 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.813459 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.813467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.813695 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.813746 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.813848 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.813960 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":547,"threads_started":1}
I20260927 04:59:43.814941 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.803772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.20.169.124:45259)
0927 04:59:43.805107 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.805112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.805128 (+ 16us) server_negotiation.cc:407] Connection header received
0927 04:59:43.805186 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.805189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.805251 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.805381 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:43.806151 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.809005 (+ 2854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.811079 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.813048 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.813288 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.813575 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.814293 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.814476 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1040,"threads_started":1}
I20260927 04:59:43.815824 28207 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 04:59:43.816509 28235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99fa63b96cd341f9b51f24e007bff59d. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:43.816782 28235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:59:43.854945 28175 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:41387
I20260927 04:59:43.855082 28379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:41387 every 8 connection(s)
I20260927 04:59:43.856640 28175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:43.863132 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 28175
I20260927 04:59:43.863463 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27599
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485183, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.124@KRBTEST.COM
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485183, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485183, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
Sep 27 04:59:43 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.66@KRBTEST.COM
I20260927 04:59:43.899850 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.860981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:54397 (local address 127.20.169.124:45259)
0927 04:59:43.861248 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.861259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.879657 (+ 18398us) server_negotiation.cc:407] Connection header received
0927 04:59:43.879753 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.879761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.879866 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.880004 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.881186 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.882264 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.885784 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.886504 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.886608 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.892012 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.892019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.892025 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.892074 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.895607 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.896698 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.896705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.896711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.896890 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.897571 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.897578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.897582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.897972 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.898113 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.899150 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.899411 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 04:59:43.901144 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.859742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:40243 (local address 127.20.169.125:45003)
0927 04:59:43.876126 (+ 16384us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.876137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.876169 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:59:43.878081 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.878088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.878199 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.878319 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.880411 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.881597 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.886532 (+ 4935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.887182 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.887293 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.892762 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.892768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.892773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.892848 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.897401 (+ 4553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.898189 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.898194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.898198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.898373 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.898845 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.898851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.898856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.899210 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.899373 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.900503 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.900728 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16205,"thread_start_us":16406,"threads_started":1}
I20260927 04:59:43.901989 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.876272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:41741 (local address 127.20.169.126:36927)
0927 04:59:43.882841 (+ 6569us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.882849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.882877 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:59:43.882939 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.882945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.883022 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.883142 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.885545 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.886736 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.889884 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.890537 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.890679 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.895211 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.895217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.895222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.895261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.898870 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.899689 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.899695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.899698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.899897 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.900380 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.900386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.900389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.900711 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.900910 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.901296 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.901523 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6394,"thread_start_us":6251,"threads_started":1}
I20260927 04:59:43.903103 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.870801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:41741)
0927 04:59:43.877416 (+ 6615us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.877434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.877554 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.883156 (+ 5602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.883161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.883650 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.885327 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.885347 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.886908 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.886919 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.889233 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.889249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.890749 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.890755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.891011 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.891708 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.891733 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.894993 (+ 3260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.899033 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.899037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.899040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.899521 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.900058 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.900062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.900065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.900234 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.900871 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.900876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.901029 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.901172 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.902158 (+ 986us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6494,"mutex_wait_us":1819,"spinlock_wait_cycles":26368,"thread_start_us":6877,"threads_started":1}
I20260927 04:59:43.903124 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.860390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.20.169.65:54397)
0927 04:59:43.876650 (+ 16260us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.876683 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.877519 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.880056 (+ 2537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.880061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.880691 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.880996 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.881008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.882492 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.882500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.885587 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.885601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.886978 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.886985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.887417 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.888290 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.888334 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.891739 (+ 3405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.895783 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.895795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.895821 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.896550 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.897068 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.897074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.897079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.897419 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.898215 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.898224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.898677 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.900344 (+ 1667us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.901831 (+ 1487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16067,"mutex_wait_us":1645,"thread_start_us":15433,"threads_started":1}
I20260927 04:59:43.903272 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.860390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.20.169.65:40243)
0927 04:59:43.870159 (+ 9769us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:43.870218 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:43.877899 (+ 7681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:43.878747 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:43.878755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:43.879453 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:43.880145 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.880180 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.883406 (+ 3226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.883412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.886347 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:43.886379 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.887557 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.887564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:43.887827 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.888431 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:43.888454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:43.892513 (+ 4059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:43.897551 (+ 5038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.897556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.897560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.898034 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.898506 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:43.898511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:43.898514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:43.898698 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:43.899442 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:43.899448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:43.899656 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 04:59:43.899820 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 04:59:43.901682 (+ 1862us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9410,"mutex_wait_us":1414,"spinlock_wait_cycles":4480,"thread_start_us":9125,"threads_started":1}
I20260927 04:59:43.905009 28381 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36927
I20260927 04:59:43.905210 28380 heartbeater.cc:347] Connected to a master server at 127.20.169.124:45259
I20260927 04:59:43.905382 28381 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:43.905387 28380 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:43.905727 28382 heartbeater.cc:347] Connected to a master server at 127.20.169.125:45003
I20260927 04:59:43.905865 28382 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:43.906414 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:43.864544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.20.169.66:36259)
0927 04:59:43.878399 (+ 13855us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:43.878408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:43.882684 (+ 4276us) server_negotiation.cc:407] Connection header received
0927 04:59:43.882754 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:43.882759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:43.882940 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:43.883121 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:43.885560 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.887131 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.889125 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:43.891379 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:43.891518 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:43.898416 (+ 6898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:43.898433 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:43.898452 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:43.898503 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:43.902221 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.903081 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.903092 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.903097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.903429 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:43.904238 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:43.904245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:43.904254 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:43.904752 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:43.905080 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:43.905823 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 04:59:43.906081 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13653,"thread_start_us":15590,"threads_started":1}
I20260927 04:59:43.906653 28382 heartbeater.cc:511] Master 127.20.169.125:45003 requested a full tablet report, sending...
I20260927 04:59:43.907872 27723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:43.908411 28381 heartbeater.cc:511] Master 127.20.169.126:36927 requested a full tablet report, sending...
I20260927 04:59:43.908579 28380 heartbeater.cc:511] Master 127.20.169.124:45259 requested a full tablet report, sending...
I20260927 04:59:43.909528 28006 ts_manager.cc:194] Registered new tserver with Master: f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387)
I20260927 04:59:43.910370 27403 ts_manager.cc:194] Re-registered known tserver with Master: f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387)
I20260927 04:59:43.911228 27403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:41741
I20260927 04:59:43.912043 28197 ts_manager.cc:194] Registered new tserver with Master: f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387)
I20260927 04:59:44.006649 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27599
I20260927 04:59:44.025300 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:36259
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:36927
--webserver_port=45891
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:59:44.305135 28271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: NON_VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: true } } }
I20260927 04:59:44.305627 28271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:44.306936 28394 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: attempting to promote NON_VOTER f6c4aae6ac914b81b1ce0d683ef12183 to VOTER
I20260927 04:59:44.307723 28394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } }
I20260927 04:59:44.310127 28207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 LEARNER]: Refusing update from remote peer 99fa63b96cd341f9b51f24e007bff59d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 04:59:44.310355 28016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Refusing update from remote peer 99fa63b96cd341f9b51f24e007bff59d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 04:59:44.310678 28271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 04:59:44.311102 28393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 04:59:44.314584 28271 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.319562 28271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.319954 28271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:44.319895 28016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.320281 28271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.320554 28271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:44.325008 28399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.325429 28399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:59:44.325506 28394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.325738 28394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:44.326426 28207 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.329767 28401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99fa63b96cd341f9b51f24e007bff59d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:44.330171 28401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:59:44.365547 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:44.365803 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:44.365898 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:44.373049 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:44.373165 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:44.373359 28392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:44.383059 28392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:36259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=45891
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:44.384346 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:44.385661 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:44.398800 28406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:44.398833 28407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:44.398888 28409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:44.399667 28392 server_base.cc:1034] running on GCE node
I20260927 04:59:44.400319 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:44.401250 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:44.402453 28392 hybrid_clock.cc:648] HybridClock initialized: now 1790485184402390 us; error 52 us; skew 500 ppm
I20260927 04:59:44.402774 28392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:44 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485184, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:44.406028 28392 init.cc:377] Logged in from keytab as oryx/127.20.169.66@KRBTEST.COM (short username oryx)
I20260927 04:59:44.408036 28392 webserver.cc:460] Webserver started at http://127.20.169.66:45891/ using document root <none> and password file <none>
I20260927 04:59:44.408751 28392 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:44.408931 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:44.414191 28392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 04:59:44.416788 28416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:44.418149 28392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:44.418309 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "ab7a27d683cb4d51884bfb576d629c97"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "de1f3e4ce44036c750e590bddf670595"
server_key_iv: "33601e7509f31c021a0ad20e4de26c86"
server_key_version: "encryptionkey@0"
I20260927 04:59:44.419257 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:44.437826 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:44.442269 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:44.442533 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:44.443461 28392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:44.444875 28392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:44.444990 28392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:44.445140 28392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:44.445242 28392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:44.480430 28392 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:36259
I20260927 04:59:44.480556 28529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:36259 every 8 connection(s)
I20260927 04:59:44.481920 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:44.486495 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 28392
I20260927 04:59:44.486796 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27733
Sep 27 04:59:44 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.67@KRBTEST.COM
Sep 27 04:59:44 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485184, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
Sep 27 04:59:44 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485184, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.124@KRBTEST.COM
I20260927 04:59:44.530761 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.488306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.20.169.67:41045)
0927 04:59:44.502030 (+ 13724us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:44.502041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:44.503083 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 04:59:44.503181 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:44.503193 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:44.503558 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:44.503756 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:44.507925 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.510659 (+ 2734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.513194 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.514628 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.514775 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.522513 (+ 7738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:44.522524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:44.522536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:44.522612 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:44.527435 (+ 4823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.528318 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.528326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.528330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.528486 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.529004 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.529010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.529015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.529459 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:44.529760 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:44.530088 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 04:59:44.530346 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13475,"thread_start_us":13223,"threads_started":1}
Sep 27 04:59:44 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485184, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
I20260927 04:59:44.532598 27857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:44.539323 28535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.484943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:46971 (local address 127.20.169.124:45259)
0927 04:59:44.511543 (+ 26600us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:44.511548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:44.511565 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:44.515952 (+ 4387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:44.515960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:44.516058 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:44.516143 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:44.520157 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.521167 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.525420 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.526090 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.526202 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.531531 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:44.531537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:44.531542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:44.531579 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:44.535185 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.536400 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.536409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.536415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.536588 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.537214 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.537219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.537223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.537593 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:44.537726 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:44.538695 (+ 969us) server_negotiation.cc:299] Negotiation successful
0927 04:59:44.538914 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26484,"thread_start_us":26274,"threads_started":1}
I20260927 04:59:44.539341 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.484864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:46935 (local address 127.20.169.126:36927)
0927 04:59:44.497236 (+ 12372us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:44.497244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:44.513577 (+ 16333us) server_negotiation.cc:407] Connection header received
0927 04:59:44.515946 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:44.515953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:44.516053 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:44.516217 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:44.520070 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.521404 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.524891 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.525551 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.525648 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.531298 (+ 5650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:44.531307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:44.531314 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:44.531378 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:44.535045 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.536421 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.536437 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.536444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.536611 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.537228 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.537236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.537242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.537627 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:44.537764 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:44.538703 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 04:59:44.538944 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12217,"thread_start_us":12375,"threads_started":1}
I20260927 04:59:44.541760 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.485494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:46935)
0927 04:59:44.513463 (+ 27969us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:44.513499 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:44.515555 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:44.516557 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:44.516571 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:44.518380 (+ 1809us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:44.519877 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:44.519907 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.522491 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.522496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:44.524684 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:44.524699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.525969 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.525976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:44.526364 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.527384 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:44.527411 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:44.530783 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:44.535215 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:44.535227 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:44.535245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:44.536252 (+ 1007us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:44.536762 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:44.536770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:44.536775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:44.537092 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:44.537841 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:44.537846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:44.538247 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 04:59:44.540067 (+ 1820us) client_negotiation.cc:240] Negotiation successful
0927 04:59:44.540427 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27758,"mutex_wait_us":399,"spinlock_wait_cycles":166016,"thread_start_us":15200,"threads_started":1}
I20260927 04:59:44.541735 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.485494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.20.169.66:46971)
0927 04:59:44.497307 (+ 11813us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:44.497387 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:44.515555 (+ 18168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:44.516557 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:44.516571 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:44.518744 (+ 2173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:44.519877 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:44.519907 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.521385 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.521393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:44.525210 (+ 3817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:44.525226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.526540 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.526546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:44.526785 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.527461 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:44.527493 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:44.531260 (+ 3767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:44.535343 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:44.535352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:44.535358 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:44.536205 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:44.536759 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:44.536767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:44.536773 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:44.537088 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:44.537816 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:44.537824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:44.538268 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 04:59:44.539530 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0927 04:59:44.540166 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11403,"mutex_wait_us":141,"spinlock_wait_cycles":14464,"thread_start_us":28998,"threads_started":1}
I20260927 04:59:44.543058 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.508463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.20.169.66:56405)
0927 04:59:44.515132 (+ 6669us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:44.515179 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:44.515594 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:44.517114 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:44.517119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:44.518386 (+ 1267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:44.521107 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:44.521126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.522700 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.522718 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:44.524684 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:44.524703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.525969 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.525976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:44.526359 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.527357 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:44.527393 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:44.531702 (+ 4309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:44.539927 (+ 8225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:44.539931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:44.539934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:44.540475 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:44.541109 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:44.541114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:44.541117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:44.541324 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:44.542056 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:44.542061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:44.542195 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 04:59:44.542359 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 04:59:44.542633 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6467,"mutex_wait_us":276,"spinlock_wait_cycles":70272,"thread_start_us":4270,"threads_started":1}
I20260927 04:59:44.543776 28530 heartbeater.cc:347] Connected to a master server at 127.20.169.124:45259
I20260927 04:59:44.543946 28531 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36927
I20260927 04:59:44.544102 28530 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:44.544453 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:44.502407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:56405 (local address 127.20.169.125:45003)
0927 04:59:44.514373 (+ 11966us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:44.514379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:44.515255 (+ 876us) server_negotiation.cc:407] Connection header received
0927 04:59:44.516586 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:44.516594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:44.516712 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:44.517440 (+ 728us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:44.521316 (+ 3876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.522485 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.524892 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:44.525552 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:44.525661 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:44.531932 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:44.531938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:44.531944 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:44.531990 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:44.539780 (+ 7790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.540633 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.540638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.540642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.540977 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:44.541540 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:44.541545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:44.541547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:44.541929 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:44.542087 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:44.543919 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0927 04:59:44.544138 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11799,"thread_start_us":11566,"threads_started":1}
I20260927 04:59:44.545014 28530 heartbeater.cc:511] Master 127.20.169.124:45259 requested a full tablet report, sending...
I20260927 04:59:44.545238 28532 heartbeater.cc:347] Connected to a master server at 127.20.169.125:45003
I20260927 04:59:44.545356 28532 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:44.545868 28532 heartbeater.cc:511] Master 127.20.169.125:45003 requested a full tablet report, sending...
I20260927 04:59:44.544111 28531 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:44.546456 28531 heartbeater.cc:511] Master 127.20.169.126:36927 requested a full tablet report, sending...
I20260927 04:59:44.546841 28197 ts_manager.cc:194] Registered new tserver with Master: ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
I20260927 04:59:44.547711 28006 ts_manager.cc:194] Registered new tserver with Master: ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
I20260927 04:59:44.547849 27403 ts_manager.cc:194] Re-registered known tserver with Master: ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
I20260927 04:59:44.548627 27403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:46935
I20260927 04:59:44.627369 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27733
I20260927 04:59:44.644752 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:41045
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:36927
--webserver_port=42853
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:59:44.913515 28381 heartbeater.cc:503] Master 127.20.169.126:36927 was elected leader, sending a full tablet report...
W20260927 04:59:44.980273 28543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:44.980544 28543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:44.980635 28543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:44.987823 28543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:44.987918 28543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:44.988026 28543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:44.997916 28543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:41045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=42853
--tserver_master_addrs=127.20.169.126:36927,127.20.169.125:45003,127.20.169.124:45259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:44.999147 28543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:45.000466 28543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:45.014001 28549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:45.014168 28548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:45.013964 28551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:45.015013 28543 server_base.cc:1034] running on GCE node
I20260927 04:59:45.015614 28543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:45.016561 28543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:45.017772 28543 hybrid_clock.cc:648] HybridClock initialized: now 1790485185017713 us; error 50 us; skew 500 ppm
I20260927 04:59:45.018083 28543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:45 dist-test-slave-39f2 krb5kdc[27358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485185, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:45.021608 28543 init.cc:377] Logged in from keytab as oryx/127.20.169.67@KRBTEST.COM (short username oryx)
I20260927 04:59:45.023306 28543 webserver.cc:460] Webserver started at http://127.20.169.67:42853/ using document root <none> and password file <none>
I20260927 04:59:45.024072 28543 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:45.024228 28543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:45.030050 28543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 04:59:45.032709 28558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:45.033998 28543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 04:59:45.034168 28543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "1769503332dc41ee90ab181cfec04f8f"
format_stamp: "Formatted at 2026-09-27 04:59:41 on dist-test-slave-39f2"
server_key: "a94254d8ca45fbe7186863da0d4ab08b"
server_key_iv: "b4b37e2439b1d514a650a4f894d48e91"
server_key_version: "encryptionkey@0"
I20260927 04:59:45.035176 28543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:45.068344 28543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:45.072664 28543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:45.072989 28543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:45.074083 28543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:45.075528 28543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:45.075640 28543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:45.075754 28543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:45.075832 28543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:45.109143 28543 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:41045
I20260927 04:59:45.109187 28671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:41045 every 8 connection(s)
I20260927 04:59:45.110594 28543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:45.116356 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 28543
Sep 27 04:59:45 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485185, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.124@KRBTEST.COM
Sep 27 04:59:45 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485185, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
Sep 27 04:59:45 dist-test-slave-39f2 krb5kdc[27358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485185, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.125@KRBTEST.COM
I20260927 04:59:45.157997 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.113665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:40107 (local address 127.20.169.124:45259)
0927 04:59:45.114169 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.114175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.115547 (+ 1372us) server_negotiation.cc:407] Connection header received
0927 04:59:45.119443 (+ 3896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.119451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.119583 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.119679 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.124038 (+ 4359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.125611 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.129297 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.130213 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.130338 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.145926 (+ 15588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.145937 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.145945 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.146030 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.152001 (+ 5971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.153285 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.153299 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.153303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.153517 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.154366 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.154372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.154375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.154864 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.155006 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.157009 (+ 2003us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.157288 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":446,"threads_started":1}
I20260927 04:59:45.160773 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.114006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:44029 (local address 127.20.169.126:36927)
0927 04:59:45.114558 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.114564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.115546 (+ 982us) server_negotiation.cc:407] Connection header received
0927 04:59:45.119289 (+ 3743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.119298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.119418 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.119570 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.129274 (+ 9704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.130757 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.133696 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.134580 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.134698 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.140994 (+ 6296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.141000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.141005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.141052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.148040 (+ 6988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.153412 (+ 5372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.153422 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.153428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.153723 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.154377 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.154385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.154391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.154915 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.155082 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.159778 (+ 4696us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.160099 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":323,"threads_started":1}
I20260927 04:59:45.162835 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.116207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:52319 (local address 127.20.169.125:45003)
0927 04:59:45.135398 (+ 19191us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.135407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.135433 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:59:45.135510 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.135518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.135610 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.135859 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.137136 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.138355 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.141038 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.142029 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.142191 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.152400 (+ 10209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.152409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.152417 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.152509 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.158481 (+ 5972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.159769 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.159777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.159783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.160092 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.160766 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.160779 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.160783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.161344 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.161511 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.162076 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.162377 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19032,"thread_start_us":19500,"threads_started":1}
I20260927 04:59:45.166917 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.114385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.20.169.67:40107)
0927 04:59:45.115391 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.115449 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.118842 (+ 3393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.122536 (+ 3694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.122543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.123184 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:45.123759 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.123791 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.126168 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.126176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.128937 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.128957 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.130862 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.133705 (+ 2843us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.134127 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.135823 (+ 1696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:45.135871 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:45.144604 (+ 8733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:45.152173 (+ 7569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:45.152181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:45.152187 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:45.153109 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:45.153725 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:45.153731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:45.153736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:45.154191 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:45.155179 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:45.155190 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:45.156668 (+ 1478us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.158647 (+ 1979us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.164682 (+ 6035us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":1183,"spinlock_wait_cycles":28288,"thread_start_us":424,"threads_started":1}
I20260927 04:59:45.168908 28672 heartbeater.cc:347] Connected to a master server at 127.20.169.124:45259
I20260927 04:59:45.169173 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.114386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:44029)
0927 04:59:45.115390 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.115447 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.119150 (+ 3703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.120203 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.120219 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.121399 (+ 1180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:45.129104 (+ 7705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.129117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.131003 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.131013 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.133533 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.133554 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.135284 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.135288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.135440 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.136488 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:45.136511 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:45.140603 (+ 4092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:45.150594 (+ 9991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:45.150609 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:45.150635 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:45.153237 (+ 2602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:45.153926 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:45.153935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:45.153941 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:45.154218 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:45.155185 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:45.155192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:45.155888 (+ 696us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.161397 (+ 5509us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.168225 (+ 6828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":6373,"spinlock_wait_cycles":39936,"thread_start_us":345,"threads_started":1}
I20260927 04:59:45.169245 28672 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:45.169808 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.115940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.20.169.67:52319)
0927 04:59:45.135006 (+ 19066us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.135022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.135140 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.135893 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.135898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.136506 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:45.136948 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.136967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.138615 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.138623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.140827 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.140845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.142665 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.142674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.143042 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.143774 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:45.143802 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:45.152115 (+ 8313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:45.158692 (+ 6577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:45.158700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:45.158706 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:45.159558 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:45.160281 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:45.160289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:45.160295 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:45.160569 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:45.161589 (+ 1020us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:45.161594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:45.161768 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.163469 (+ 1701us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.168967 (+ 5498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18941,"mutex_wait_us":5116,"thread_start_us":18785,"threads_started":1}
I20260927 04:59:45.170207 28672 heartbeater.cc:511] Master 127.20.169.124:45259 requested a full tablet report, sending...
I20260927 04:59:45.171042 28674 heartbeater.cc:347] Connected to a master server at 127.20.169.125:45003
I20260927 04:59:45.171077 28673 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36927
I20260927 04:59:45.171166 28673 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:45.171597 28673 heartbeater.cc:511] Master 127.20.169.126:36927 requested a full tablet report, sending...
I20260927 04:59:45.171170 28674 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:45.172209 28674 heartbeater.cc:511] Master 127.20.169.125:45003 requested a full tablet report, sending...
I20260927 04:59:45.173169 27403 ts_manager.cc:194] Re-registered known tserver with Master: 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045)
I20260927 04:59:45.173874 28006 ts_manager.cc:194] Registered new tserver with Master: 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045)
I20260927 04:59:45.174008 28197 ts_manager.cc:194] Registered new tserver with Master: 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045)
I20260927 04:59:45.174623 27403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:44029
I20260927 04:59:45.188202 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.137790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.20.169.126:36927)
0927 04:59:45.161509 (+ 23719us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.161515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.161533 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:59:45.161581 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.161585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.161691 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.161794 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.164915 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.166148 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.170759 (+ 4611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.171912 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.172049 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.180398 (+ 8349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.180405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.180411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.180476 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.184423 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.185215 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.185221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.185225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.185525 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.186197 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.186202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.186206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.186609 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.186874 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.187463 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.187691 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23413,"thread_start_us":23508,"threads_started":1}
I20260927 04:59:45.191520 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.141307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.20.169.124:45259)
0927 04:59:45.155278 (+ 13971us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.155286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.164314 (+ 9028us) server_negotiation.cc:407] Connection header received
0927 04:59:45.164582 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.164591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.164671 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.164748 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.167530 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.168664 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.175254 (+ 6590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.175934 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.176031 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.179629 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.179638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.179643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.179688 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.185224 (+ 5536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.185998 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.186006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.186011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.186285 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.187921 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.187928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.187931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.188459 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.188857 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.190474 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.190794 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9346,"thread_start_us":9113,"threads_started":1}
I20260927 04:59:45.193507 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.157238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.20.169.125:45003)
0927 04:59:45.163103 (+ 5865us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.163108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.170238 (+ 7130us) server_negotiation.cc:407] Connection header received
0927 04:59:45.170560 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.170565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.170665 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.170733 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.174259 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.175411 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.177661 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.178400 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.178514 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.182074 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.182084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.182093 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.182150 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.187387 (+ 5237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.188523 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.188532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.188537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.188843 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.189638 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.189646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.189651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.190161 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.190614 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.192623 (+ 2009us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.192964 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5740,"thread_start_us":7121,"threads_started":1}
I20260927 04:59:45.193310 27403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:45.203279 27403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:45.256513 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.226446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:41045 (local address 127.0.0.1:39796)
0927 04:59:45.241991 (+ 15545us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.242018 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.242283 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.242980 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.242986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.243021 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:45.243934 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.243949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.247451 (+ 3502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.247459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.252619 (+ 5160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.252634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.255250 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.255261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.255530 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.255563 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.255660 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.255978 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":793,"threads_started":1}
I20260927 04:59:45.257553 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.227197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.20.169.67:41045)
0927 04:59:45.235001 (+ 7804us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.235008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.242186 (+ 7178us) server_negotiation.cc:407] Connection header received
0927 04:59:45.242462 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.242469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.242729 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.242998 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:45.244244 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.246852 (+ 2608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.252901 (+ 6049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.254451 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.254602 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.255114 (+ 512us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.256872 (+ 1758us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.257095 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7550,"thread_start_us":7132,"threads_started":1}
I20260927 04:59:45.261145 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.228454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.20.169.65:41387)
0927 04:59:45.241988 (+ 13534us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.241996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.249144 (+ 7148us) server_negotiation.cc:407] Connection header received
0927 04:59:45.249376 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.249383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.249639 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.249910 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:45.251625 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.252912 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.256880 (+ 3968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.258922 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.259060 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.259358 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.260283 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.260576 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13289,"thread_start_us":13523,"threads_started":1}
I20260927 04:59:45.261296 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.228411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.20.169.66:36259)
0927 04:59:45.243807 (+ 15396us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.243818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.246158 (+ 2340us) server_negotiation.cc:407] Connection header received
0927 04:59:45.246292 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.246304 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.246635 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.246900 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:45.249430 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.251997 (+ 2567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.256110 (+ 4113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.259567 (+ 3457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.259737 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.260044 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.260614 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.260858 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15190,"thread_start_us":15353,"threads_started":1}
I20260927 04:59:45.262050 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.227744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:36259 (local address 127.0.0.1:34834)
0927 04:59:45.245766 (+ 18022us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.245804 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.245993 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.246877 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.246882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.246920 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:45.249157 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.249176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.252156 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.252162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.255875 (+ 3713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.255895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.259879 (+ 3984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.259885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.260129 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.260160 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.260254 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.260414 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17804,"mutex_wait_us":44,"thread_start_us":20543,"threads_started":1}
I20260927 04:59:45.262231 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.227741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41387 (local address 127.0.0.1:51346)
0927 04:59:45.249017 (+ 21276us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.249032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.249211 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.249910 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.249915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.249938 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:45.251278 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.251297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.253093 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.253098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.256692 (+ 3594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.256705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.259373 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.259387 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.259702 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.259746 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.259939 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.261487 (+ 1548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21048,"mutex_wait_us":322,"thread_start_us":17270,"threads_started":1}
I20260927 04:59:45.264400 28606 tablet_service.cc:1467] Processing CreateTablet for tablet 1c99886eac354e11a78146832731dd5d (DEFAULT_TABLE table=test-table [id=b67aeefc0f5f4911b3331d210f34d872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:45.265800 28314 tablet_service.cc:1467] Processing CreateTablet for tablet 1c99886eac354e11a78146832731dd5d (DEFAULT_TABLE table=test-table [id=b67aeefc0f5f4911b3331d210f34d872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:45.266383 28606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c99886eac354e11a78146832731dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:45.266666 28465 tablet_service.cc:1467] Processing CreateTablet for tablet 1c99886eac354e11a78146832731dd5d (DEFAULT_TABLE table=test-table [id=b67aeefc0f5f4911b3331d210f34d872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:45.267863 28314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c99886eac354e11a78146832731dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:45.268244 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c99886eac354e11a78146832731dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:45.286125 28706 tablet_bootstrap.cc:492] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Bootstrap starting.
I20260927 04:59:45.289086 28707 tablet_bootstrap.cc:492] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Bootstrap starting.
I20260927 04:59:45.290699 28708 tablet_bootstrap.cc:492] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Bootstrap starting.
I20260927 04:59:45.292994 28706 tablet_bootstrap.cc:654] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:45.294431 28707 tablet_bootstrap.cc:654] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:45.294417 28706 log.cc:824] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:45.294624 28708 tablet_bootstrap.cc:654] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:45.295774 28708 log.cc:824] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:45.295992 28707 log.cc:824] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:45.299100 28707 tablet_bootstrap.cc:492] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: No bootstrap required, opened a new log
I20260927 04:59:45.299312 28708 tablet_bootstrap.cc:492] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: No bootstrap required, opened a new log
I20260927 04:59:45.299377 28707 ts_tablet_manager.cc:1397] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 04:59:45.299578 28708 ts_tablet_manager.cc:1397] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 04:59:45.301203 28706 tablet_bootstrap.cc:492] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: No bootstrap required, opened a new log
I20260927 04:59:45.301542 28706 ts_tablet_manager.cc:1397] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260927 04:59:45.310272 28708 raft_consensus.cc:348] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.310807 28708 raft_consensus.cc:374] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:45.310983 28708 raft_consensus.cc:729] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7a27d683cb4d51884bfb576d629c97, State: Initialized, Role: FOLLOWER
I20260927 04:59:45.311725 28708 consensus_queue.cc:260] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.314051 28708 ts_tablet_manager.cc:1428] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260927 04:59:45.314527 28531 heartbeater.cc:503] Master 127.20.169.126:36927 was elected leader, sending a full tablet report...
I20260927 04:59:45.318835 28707 raft_consensus.cc:348] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.319130 28706 raft_consensus.cc:348] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.319681 28707 raft_consensus.cc:374] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:45.319787 28706 raft_consensus.cc:374] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:45.319976 28706 raft_consensus.cc:729] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f245a22d06014d6d9f47477d25dcd6fc, State: Initialized, Role: FOLLOWER
I20260927 04:59:45.319932 28707 raft_consensus.cc:729] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1769503332dc41ee90ab181cfec04f8f, State: Initialized, Role: FOLLOWER
I20260927 04:59:45.320880 28706 consensus_queue.cc:260] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.321040 28707 consensus_queue.cc:260] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.323632 28706 ts_tablet_manager.cc:1428] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260927 04:59:45.324050 28673 heartbeater.cc:503] Master 127.20.169.126:36927 was elected leader, sending a full tablet report...
I20260927 04:59:45.326131 28707 ts_tablet_manager.cc:1428] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Time spent starting tablet: real 0.027s user 0.017s sys 0.004s
I20260927 04:59:45.340152 28713 raft_consensus.cc:482] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:45.340575 28713 raft_consensus.cc:504] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.342648 28713 leader_election.cc:288] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045), ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
I20260927 04:59:45.354270 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.342972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:45501 (local address 127.20.169.67:41045)
0927 04:59:45.343272 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.343280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.344600 (+ 1320us) server_negotiation.cc:407] Connection header received
0927 04:59:45.345043 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.345051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.345160 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.345271 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:45.346108 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.347624 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.352086 (+ 4462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.353139 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.353278 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.353337 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.353756 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.353914 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 04:59:45.354562 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.343020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:41045 (local address 127.20.169.65:45501)
0927 04:59:45.344468 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.344497 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.344770 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.345291 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.345295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.345315 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:45.345962 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.345975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.348047 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.348064 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.351904 (+ 3840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.351918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.353430 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.353435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.353600 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.353624 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.353730 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.353878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"mutex_wait_us":46,"thread_start_us":850,"threads_started":1}
I20260927 04:59:45.355535 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.343883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:36259 (local address 127.20.169.65:58953)
0927 04:59:45.345354 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:45.345386 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:45.345538 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:45.345882 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:45.345886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:45.345912 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:45.346880 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.346895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.348464 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.348472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.351905 (+ 3433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:45.351918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.354707 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.354713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:45.354969 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.355019 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 04:59:45.355149 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 04:59:45.355283 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":52,"thread_start_us":921,"threads_started":1}
I20260927 04:59:45.355667 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.343436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:58953 (local address 127.20.169.66:36259)
0927 04:59:45.343723 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.343732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.345509 (+ 1777us) server_negotiation.cc:407] Connection header received
0927 04:59:45.345712 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.345717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.345773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.345867 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:45.347089 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.348277 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.353427 (+ 5150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.354325 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.354500 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.354560 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.355175 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.355317 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 04:59:45.356118 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c99886eac354e11a78146832731dd5d" candidate_uuid: "f245a22d06014d6d9f47477d25dcd6fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1769503332dc41ee90ab181cfec04f8f" is_pre_election: true
I20260927 04:59:45.356969 28627 raft_consensus.cc:2454] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f245a22d06014d6d9f47477d25dcd6fc in term 0.
I20260927 04:59:45.357061 28485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c99886eac354e11a78146832731dd5d" candidate_uuid: "f245a22d06014d6d9f47477d25dcd6fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7a27d683cb4d51884bfb576d629c97" is_pre_election: true
I20260927 04:59:45.357705 28485 raft_consensus.cc:2454] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f245a22d06014d6d9f47477d25dcd6fc in term 0.
I20260927 04:59:45.357913 28266 leader_election.cc:302] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1769503332dc41ee90ab181cfec04f8f, f245a22d06014d6d9f47477d25dcd6fc; no voters:
I20260927 04:59:45.358403 28713 raft_consensus.cc:2781] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:45.358642 28713 raft_consensus.cc:482] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:45.358784 28713 raft_consensus.cc:3037] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:45.361248 28713 raft_consensus.cc:504] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.361819 28713 leader_election.cc:288] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [CANDIDATE]: Term 1 election: Requested vote from peers 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045), ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
I20260927 04:59:45.362262 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c99886eac354e11a78146832731dd5d" candidate_uuid: "f245a22d06014d6d9f47477d25dcd6fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1769503332dc41ee90ab181cfec04f8f"
I20260927 04:59:45.362309 28485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c99886eac354e11a78146832731dd5d" candidate_uuid: "f245a22d06014d6d9f47477d25dcd6fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7a27d683cb4d51884bfb576d629c97"
I20260927 04:59:45.362490 28627 raft_consensus.cc:3037] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:45.362491 28485 raft_consensus.cc:3037] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Advancing to term 1
W20260927 04:59:45.363312 28383 tablet.cc:2369] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:59:45.363814 28675 tablet.cc:2369] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:45.365813 28485 raft_consensus.cc:2454] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f245a22d06014d6d9f47477d25dcd6fc in term 1.
I20260927 04:59:45.365849 28627 raft_consensus.cc:2454] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f245a22d06014d6d9f47477d25dcd6fc in term 1.
I20260927 04:59:45.366262 28264 leader_election.cc:302] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7a27d683cb4d51884bfb576d629c97, f245a22d06014d6d9f47477d25dcd6fc; no voters:
I20260927 04:59:45.366597 28713 raft_consensus.cc:2781] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:45.367022 28713 raft_consensus.cc:686] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 1 LEADER]: Becoming Leader. State: Replica: f245a22d06014d6d9f47477d25dcd6fc, State: Running, Role: LEADER
I20260927 04:59:45.367492 28713 consensus_queue.cc:237] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:45.373109 27403 catalog_manager.cc:5495] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc reported cstate change: term changed from 0 to 1, leader changed from <none> to f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65). New cstate: current_term: 1 leader_uuid: "f245a22d06014d6d9f47477d25dcd6fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } health_report { overall_health: UNKNOWN } } }
I20260927 04:59:45.416770 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.410202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.20.169.65:41387)
0927 04:59:45.410410 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.410415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.410685 (+ 270us) server_negotiation.cc:407] Connection header received
0927 04:59:45.410882 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.410888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.410995 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.411149 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:45.411509 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.412682 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.414133 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.414835 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.414964 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.415437 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:45.416145 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:45.416296 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.416378 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.416465 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:59:45.425976 28485 raft_consensus.cc:1261] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Refusing update from remote peer f245a22d06014d6d9f47477d25dcd6fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:45.425983 28627 raft_consensus.cc:1261] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Refusing update from remote peer f245a22d06014d6d9f47477d25dcd6fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:45.426909 28717 consensus_queue.cc:1035] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:45.427290 28713 consensus_queue.cc:1035] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:45.439328 28720 mvcc.cc:205] Tried to move back new op lower bound from 7333827319494942720 to 7333827319268995072. Current Snapshot: MvccSnapshot[applied={T|T < 7333827319494942720}]
I20260927 04:59:45.440079 28721 mvcc.cc:205] Tried to move back new op lower bound from 7333827319494942720 to 7333827319268995072. Current Snapshot: MvccSnapshot[applied={T|T < 7333827319494942720}]
I20260927 04:59:45.461118 27403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:45.461427 27403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:45.470515 27403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99fa63b96cd341f9b51f24e007bff59d: Sending DeleteTablet for 3 replicas of tablet 1c99886eac354e11a78146832731dd5d
I20260927 04:59:45.471932 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c99886eac354e11a78146832731dd5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:45 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34834
I20260927 04:59:45.472952 28606 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c99886eac354e11a78146832731dd5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:45 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:39796
I20260927 04:59:45.479516 28314 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c99886eac354e11a78146832731dd5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:45 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:51346
I20260927 04:59:45.486610 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c99886eac354e11a78146832731dd5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:45 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34834
I20260927 04:59:45.494410 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27371
I20260927 04:59:45.496125 27433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:45.503686 28730 tablet_replica.cc:330] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: stopping tablet replica
I20260927 04:59:45.504359 28730 raft_consensus.cc:2229] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:45.504999 28732 tablet_replica.cc:330] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: stopping tablet replica
I20260927 04:59:45.505626 28732 raft_consensus.cc:2229] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:45.508750 28731 tablet_replica.cc:330] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: stopping tablet replica
I20260927 04:59:45.509248 28731 raft_consensus.cc:2229] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:45.509954 28732 raft_consensus.cc:2258] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:45.510237 28731 raft_consensus.cc:2258] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:45.512209 28730 raft_consensus.cc:2258] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:45.513393 28731 ts_tablet_manager.cc:1905] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:45.518618 28732 ts_tablet_manager.cc:1905] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:45.524154 28730 ts_tablet_manager.cc:1905] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:45.536489 28731 ts_tablet_manager.cc:1918] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:45.540009 28730 ts_tablet_manager.cc:1918] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:45.540319 28730 log.cc:1196] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/1c99886eac354e11a78146832731dd5d
I20260927 04:59:45.541539 28730 ts_tablet_manager.cc:1939] T 1c99886eac354e11a78146832731dd5d P ab7a27d683cb4d51884bfb576d629c97: Deleting consensus metadata
I20260927 04:59:45.546406 28731 log.cc:1196] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/1c99886eac354e11a78146832731dd5d
I20260927 04:59:45.548319 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.490892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.20.169.125:45003)
0927 04:59:45.491100 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.491107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.492936 (+ 1829us) server_negotiation.cc:407] Connection header received
0927 04:59:45.495210 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.495216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.495339 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.495456 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.521091 (+ 25635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.522756 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.526018 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.527103 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.527224 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.537099 (+ 9875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.537107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.537113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.537167 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.542986 (+ 5819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.544374 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.544380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.544386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.544653 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.545307 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.545314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.545318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.546035 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.546414 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.547418 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.547714 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 04:59:45.550264 28731 ts_tablet_manager.cc:1939] T 1c99886eac354e11a78146832731dd5d P 1769503332dc41ee90ab181cfec04f8f: Deleting consensus metadata
I20260927 04:59:45.553594 28732 ts_tablet_manager.cc:1918] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:45.553972 28732 log.cc:1196] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/1c99886eac354e11a78146832731dd5d
I20260927 04:59:45.555030 28732 ts_tablet_manager.cc:1939] T 1c99886eac354e11a78146832731dd5d P f245a22d06014d6d9f47477d25dcd6fc: Deleting consensus metadata
I20260927 04:59:45.567590 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.493147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.20.169.124:45259)
0927 04:59:45.493467 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.493476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.551648 (+ 58172us) server_negotiation.cc:407] Connection header received
0927 04:59:45.552009 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.552015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.552113 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.552184 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.555268 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.556893 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.559050 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.559631 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.559773 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.562399 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.562405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.562414 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.562449 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.565066 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.565686 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.565690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.565692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.565817 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.566184 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.566189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.566192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.566465 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.566715 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.567016 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.567239 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 04:59:45.676862 27390 catalog_manager.cc:4841] TS f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387): tablet 1c99886eac354e11a78146832731dd5d (table test-table [id=b67aeefc0f5f4911b3331d210f34d872]) successfully deleted
I20260927 04:59:45.676898 27389 catalog_manager.cc:4841] TS ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259): tablet 1c99886eac354e11a78146832731dd5d (table test-table [id=b67aeefc0f5f4911b3331d210f34d872]) successfully deleted
I20260927 04:59:45.677197 27390 catalog_manager.cc:4841] TS 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045): tablet 1c99886eac354e11a78146832731dd5d (table test-table [id=b67aeefc0f5f4911b3331d210f34d872]) successfully deleted
W20260927 04:59:45.677517 27389 catalog_manager.cc:4808] TS ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259): delete failed for tablet 1c99886eac354e11a78146832731dd5d because the tablet was not found. No further retry: Not found: Tablet not found: 1c99886eac354e11a78146832731dd5d
I20260927 04:59:45.677747 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27371
I20260927 04:59:45.737254 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.719746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.20.169.124:45259)
0927 04:59:45.720032 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.720041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.721519 (+ 1478us) server_negotiation.cc:407] Connection header received
0927 04:59:45.721841 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.721849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.721991 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.722092 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.724839 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.725983 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.727746 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.728210 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.728335 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.732042 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.732050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.732056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.732103 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.734698 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.735381 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.735387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.735391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.735515 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.735871 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.735876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.735879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.736159 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.736416 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.736738 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.736981 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 04:59:45.738131 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:45.721209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.20.169.125:45003)
0927 04:59:45.721409 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:45.721415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:45.722273 (+ 858us) server_negotiation.cc:407] Connection header received
0927 04:59:45.722426 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:45.722432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:45.722517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:45.722620 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:45.724904 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.725603 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.727385 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:45.728078 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:45.728214 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:45.731343 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:45.731349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:45.731354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:45.731396 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:45.734727 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.735563 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.735571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.735574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.735717 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:45.736162 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:45.736168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:45.736171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:45.736522 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:45.736722 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:45.737482 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 04:59:45.737732 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 04:59:46.453567 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.452881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:60147)
0927 04:59:46.453260 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.453376 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":185,"threads_started":1}
W20260927 04:59:46.453990 28673 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:36927 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.455935 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.454959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:50283)
0927 04:59:46.455557 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.455698 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":382,"threads_started":1}
W20260927 04:59:46.456359 28381 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:36927 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.473333 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.472454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:44649)
0927 04:59:46.472990 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.473132 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":240,"threads_started":1}
W20260927 04:59:46.473757 28531 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:36927 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.969488 28744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99fa63b96cd341f9b51f24e007bff59d)
I20260927 04:59:46.969805 28744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } }
I20260927 04:59:46.970642 28745 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 99fa63b96cd341f9b51f24e007bff59d)
I20260927 04:59:46.970889 28745 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } }
I20260927 04:59:46.971412 28744 leader_election.cc:288] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927), f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124:45259)
I20260927 04:59:46.973076 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.971534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49346)
0927 04:59:46.972329 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.972562 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":485,"threads_started":1}
W20260927 04:59:46.973351 27990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.973483 28745 leader_election.cc:288] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927), 3543ce954bc14f5f898b6106cbd69262 (127.20.169.125:45003)
I20260927 04:59:46.974702 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.973570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49350)
0927 04:59:46.974286 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.974446 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":268,"threads_started":1}
W20260927 04:59:46.976572 28181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.980647 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.977848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49370)
0927 04:59:46.980046 (+ 2198us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.980454 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2093,"threads_started":1}
I20260927 04:59:46.986402 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.971993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.20.169.124:45259)
0927 04:59:46.972580 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:46.972586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:46.973230 (+ 644us) server_negotiation.cc:407] Connection header received
0927 04:59:46.973530 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:46.973535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:46.973608 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:46.973735 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:46.977916 (+ 4181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.979623 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.982918 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.984869 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.985034 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:46.985142 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:46.985937 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 04:59:46.986066 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":311,"threads_started":1}
W20260927 04:59:46.986339 27990 leader_election.cc:334] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927): Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.987294 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.971690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:45259 (local address 127.0.0.1:38364)
0927 04:59:46.973132 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.973150 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:46.973344 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:46.974094 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:46.974102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:46.974138 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:46.977691 (+ 3553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:46.977706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.979863 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.979872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:46.982639 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:46.982660 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.985359 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.985367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:46.985729 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:46.985771 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 04:59:46.986712 (+ 941us) client_negotiation.cc:240] Negotiation successful
0927 04:59:46.986889 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1303,"mutex_wait_us":32,"thread_start_us":1075,"threads_started":1}
I20260927 04:59:46.987576 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.974425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:45003 (local address 127.0.0.1:60640)
0927 04:59:46.974988 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.975007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:46.975137 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:46.978346 (+ 3209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:46.978354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:46.978407 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:46.978766 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:46.978780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.981052 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.981061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:46.983852 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:46.983872 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.986734 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.986744 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:46.986983 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:46.987024 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 04:59:46.987098 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 04:59:46.987267 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":1844,"threads_started":1}
I20260927 04:59:46.987998 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3543ce954bc14f5f898b6106cbd69262" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" is_pre_election: true
I20260927 04:59:46.988272 28752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.974486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.20.169.125:45003)
0927 04:59:46.977694 (+ 3208us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:46.977703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:46.977735 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:59:46.977810 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:46.977818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:46.977903 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:46.978032 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:46.979001 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.980847 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.984134 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:46.986466 (+ 2332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:46.987705 (+ 1239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:46.987768 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:46.987818 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 04:59:46.987943 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":428,"server-negotiator.queue_time_us":3003,"thread_start_us":2786,"threads_started":1}
I20260927 04:59:46.988404 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.987999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49386)
0927 04:59:46.988125 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.988215 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:59:46.988371 28207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3543ce954bc14f5f898b6106cbd69262 in term 1.
W20260927 04:59:46.988835 28181 leader_election.cc:334] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927): Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.988935 27991 leader_election.cc:302] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3543ce954bc14f5f898b6106cbd69262, f6c4aae6ac914b81b1ce0d683ef12183; no voters: 99fa63b96cd341f9b51f24e007bff59d
I20260927 04:59:46.989308 28744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 04:59:46.989429 28744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 99fa63b96cd341f9b51f24e007bff59d)
I20260927 04:59:46.989549 28744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:59:46.989835 28016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3543ce954bc14f5f898b6106cbd69262" is_pre_election: true
I20260927 04:59:46.991997 28744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } }
I20260927 04:59:46.992461 28016 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6c4aae6ac914b81b1ce0d683ef12183 in current term 2: Already voted for candidate 3543ce954bc14f5f898b6106cbd69262 in this term.
I20260927 04:59:46.992841 28744 leader_election.cc:288] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [CANDIDATE]: Term 2 election: Requested vote from peers 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927), f6c4aae6ac914b81b1ce0d683ef12183 (127.20.169.124:45259)
I20260927 04:59:46.993284 28207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3543ce954bc14f5f898b6106cbd69262" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6c4aae6ac914b81b1ce0d683ef12183"
I20260927 04:59:46.993413 28181 leader_election.cc:302] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6c4aae6ac914b81b1ce0d683ef12183; no voters: 3543ce954bc14f5f898b6106cbd69262, 99fa63b96cd341f9b51f24e007bff59d
I20260927 04:59:46.993543 28207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:59:46.994023 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.993611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49394)
0927 04:59:46.993795 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.993880 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 04:59:46.995178 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:46.994741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49398)
0927 04:59:46.994915 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:46.995011 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 04:59:46.995337 27990 leader_election.cc:334] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927): Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
I20260927 04:59:46.996351 28207 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3543ce954bc14f5f898b6106cbd69262 in term 2.
I20260927 04:59:46.996570 28745 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:59:46.996945 27991 leader_election.cc:302] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3543ce954bc14f5f898b6106cbd69262, f6c4aae6ac914b81b1ce0d683ef12183; no voters: 99fa63b96cd341f9b51f24e007bff59d
I20260927 04:59:46.997212 28744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:59:46.997691 28744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [term 2 LEADER]: Becoming Leader. State: Replica: 3543ce954bc14f5f898b6106cbd69262, State: Running, Role: LEADER
I20260927 04:59:46.998065 28744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } }
I20260927 04:59:46.999790 28755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3543ce954bc14f5f898b6106cbd69262. Latest consensus state: current_term: 2 leader_uuid: "3543ce954bc14f5f898b6106cbd69262" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:47.000123 28755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:47.000783 28757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:47.004299 28757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:47.005226 28757 catalog_manager.cc:1259] Loaded cluster ID: 56befa11045343c0be45d783eb3564eb
I20260927 04:59:47.005291 28757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:47.006343 28757 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:47.007085 28757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Loaded TSK: 0
I20260927 04:59:47.007865 28757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:59:47.045343 28005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:47.056720 28005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:47.062829 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.062444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49426)
0927 04:59:47.062613 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.062702 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:59:47.062860 28207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [term 2 FOLLOWER]: Refusing update from remote peer 3543ce954bc14f5f898b6106cbd69262: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 04:59:47.063678 28755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 04:59:47.063921 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.063438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49428)
0927 04:59:47.063645 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.063739 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 04:59:47.064092 27990 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 -> Peer 99fa63b96cd341f9b51f24e007bff59d (127.20.169.126:36927): Couldn't send request to peer 99fa63b96cd341f9b51f24e007bff59d. Status: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:59:47.066558 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3543ce954bc14f5f898b6106cbd69262. Latest consensus state: current_term: 2 leader_uuid: "3543ce954bc14f5f898b6106cbd69262" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:47.066893 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:59:47.068640 28755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3543ce954bc14f5f898b6106cbd69262" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:47.068936 28755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:47.069708 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3543ce954bc14f5f898b6106cbd69262" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "99fa63b96cd341f9b51f24e007bff59d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36927 } } peers { permanent_uuid: "3543ce954bc14f5f898b6106cbd69262" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 45003 } attrs { promote: false } } peers { permanent_uuid: "f6c4aae6ac914b81b1ce0d683ef12183" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 45259 } attrs { promote: false } } }
I20260927 04:59:47.069964 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c4aae6ac914b81b1ce0d683ef12183 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:59:47.086073 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.077061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:41045 (local address 127.0.0.1:39798)
0927 04:59:47.077420 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.077440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.077545 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.077947 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.077954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.077983 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.078167 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.078178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.079559 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.079567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.083689 (+ 4122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.083707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.085187 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.085192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.085422 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.085474 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.085560 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.085724 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1341}
I20260927 04:59:47.086076 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.077167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.20.169.67:41045)
0927 04:59:47.077561 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.077566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.077582 (+ 16us) server_negotiation.cc:407] Connection header received
0927 04:59:47.077642 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.077646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.077745 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.077848 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.078309 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.079399 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.083848 (+ 4449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.085034 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.085170 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.085291 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.085626 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.085769 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":1405,"threads_started":1}
I20260927 04:59:47.087139 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.077920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:36259 (local address 127.0.0.1:34844)
0927 04:59:47.078308 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.078325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.078434 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.079172 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.079177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.079199 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.079504 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.079515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.081017 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.081025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.083533 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.083553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.085248 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.085254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.086592 (+ 1338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.086628 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.086716 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.086857 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":1121}
I20260927 04:59:47.088006 28606 tablet_service.cc:1467] Processing CreateTablet for tablet d8f43a31bb37488f9b799e5a148fad3b (DEFAULT_TABLE table=test-table [id=e1be716dee1c4b2bb980eb1b6e6bf29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:47.088394 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.078445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.20.169.66:36259)
0927 04:59:47.078853 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.078859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.078876 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:47.078953 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.078959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.079028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.079652 (+ 624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.079801 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.080802 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.083780 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.084990 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.085400 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.085500 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.087937 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.088087 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":279,"threads_started":1}
I20260927 04:59:47.088461 28606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f43a31bb37488f9b799e5a148fad3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:47.093114 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.078537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41387 (local address 127.0.0.1:51364)
0927 04:59:47.078803 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.078848 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.078939 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.085493 (+ 6554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.085497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.085525 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.085825 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.085837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.087409 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.087416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.090266 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.090284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.092305 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.092312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.092523 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.092558 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.092653 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.092791 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 04:59:47.093219 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.078114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.20.169.65:41387)
0927 04:59:47.085168 (+ 7054us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.085174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.085192 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:59:47.085260 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.085265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.085344 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.085973 (+ 629us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.086275 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.087261 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.090479 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.092063 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.092213 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.092353 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.092685 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.092891 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6888,"thread_start_us":6771,"threads_started":1}
I20260927 04:59:47.093523 28465 tablet_service.cc:1467] Processing CreateTablet for tablet d8f43a31bb37488f9b799e5a148fad3b (DEFAULT_TABLE table=test-table [id=e1be716dee1c4b2bb980eb1b6e6bf29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:47.093931 28465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f43a31bb37488f9b799e5a148fad3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:47.093925 28314 tablet_service.cc:1467] Processing CreateTablet for tablet d8f43a31bb37488f9b799e5a148fad3b (DEFAULT_TABLE table=test-table [id=e1be716dee1c4b2bb980eb1b6e6bf29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:47.094362 28314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f43a31bb37488f9b799e5a148fad3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:47.095966 28768 tablet_bootstrap.cc:492] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Bootstrap starting.
I20260927 04:59:47.097842 28768 tablet_bootstrap.cc:654] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:47.100076 28768 tablet_bootstrap.cc:492] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: No bootstrap required, opened a new log
I20260927 04:59:47.100301 28768 ts_tablet_manager.cc:1397] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 04:59:47.100359 28770 tablet_bootstrap.cc:492] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Bootstrap starting.
I20260927 04:59:47.101097 28771 tablet_bootstrap.cc:492] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Bootstrap starting.
I20260927 04:59:47.101516 28768 raft_consensus.cc:348] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.101850 28768 raft_consensus.cc:374] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:47.102020 28768 raft_consensus.cc:729] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1769503332dc41ee90ab181cfec04f8f, State: Initialized, Role: FOLLOWER
I20260927 04:59:47.102383 28768 consensus_queue.cc:260] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.102644 28770 tablet_bootstrap.cc:654] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:47.103019 28771 tablet_bootstrap.cc:654] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:47.104071 28768 ts_tablet_manager.cc:1428] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 04:59:47.106213 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.104914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:50459)
0927 04:59:47.105641 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.105836 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":287,"threads_started":1}
I20260927 04:59:47.106715 28770 tablet_bootstrap.cc:492] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: No bootstrap required, opened a new log
I20260927 04:59:47.106863 28770 ts_tablet_manager.cc:1397] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 04:59:47.107676 28770 raft_consensus.cc:348] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.107921 28770 raft_consensus.cc:374] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:47.108009 28770 raft_consensus.cc:729] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7a27d683cb4d51884bfb576d629c97, State: Initialized, Role: FOLLOWER
I20260927 04:59:47.108222 28770 consensus_queue.cc:260] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.112610 28674 heartbeater.cc:503] Master 127.20.169.125:45003 was elected leader, sending a full tablet report...
I20260927 04:59:47.112993 28771 tablet_bootstrap.cc:492] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: No bootstrap required, opened a new log
I20260927 04:59:47.113168 28771 ts_tablet_manager.cc:1397] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260927 04:59:47.113904 28771 raft_consensus.cc:348] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.114166 28771 raft_consensus.cc:374] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:47.114276 28771 raft_consensus.cc:729] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f245a22d06014d6d9f47477d25dcd6fc, State: Initialized, Role: FOLLOWER
I20260927 04:59:47.114508 28771 consensus_queue.cc:260] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.118665 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.117635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:39045)
0927 04:59:47.118360 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.118475 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570}
I20260927 04:59:47.122659 28770 ts_tablet_manager.cc:1428] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 04:59:47.126432 28771 ts_tablet_manager.cc:1428] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Time spent starting tablet: real 0.013s user 0.006s sys 0.003s
I20260927 04:59:47.129727 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.123306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:49143)
0927 04:59:47.129279 (+ 5973us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.129470 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5823,"thread_start_us":5687,"threads_started":1}
I20260927 04:59:47.130578 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.127834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:48207)
0927 04:59:47.130194 (+ 2360us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.130368 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2244,"threads_started":1}
I20260927 04:59:47.131422 28532 heartbeater.cc:503] Master 127.20.169.125:45003 was elected leader, sending a full tablet report...
I20260927 04:59:47.132000 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.131378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:51195)
0927 04:59:47.131598 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.131677 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 04:59:47.133071 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.132258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:47337)
0927 04:59:47.132862 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.132949 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483}
I20260927 04:59:47.134359 28382 heartbeater.cc:503] Master 127.20.169.125:45003 was elected leader, sending a full tablet report...
W20260927 04:59:47.254329 28533 tablet.cc:2369] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:47.377087 28776 raft_consensus.cc:482] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:47.377322 28776 raft_consensus.cc:504] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.378927 28776 leader_election.cc:288] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045), f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387)
I20260927 04:59:47.383791 28772 raft_consensus.cc:482] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:47.384058 28772 raft_consensus.cc:504] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.385769 28772 leader_election.cc:288] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387), ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259)
I20260927 04:59:47.386303 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.379264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:36141 (local address 127.20.169.67:41045)
0927 04:59:47.379437 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.379441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.379674 (+ 233us) server_negotiation.cc:407] Connection header received
0927 04:59:47.379951 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.379957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.380056 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.380185 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.380701 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.381830 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.383816 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.385164 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.385355 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.385448 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.385767 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.385954 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:59:47.387256 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.379264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:41045 (local address 127.20.169.66:36141)
0927 04:59:47.379500 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.379521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.379693 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.380219 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.380224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.380263 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.380587 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.380595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.381958 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.381963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.383624 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.383656 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.385355 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.385362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.385552 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.385577 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.386814 (+ 1237us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.386954 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":4}
I20260927 04:59:47.387461 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.379909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:58767 (local address 127.20.169.65:41387)
0927 04:59:47.380130 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.380136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.380600 (+ 464us) server_negotiation.cc:407] Connection header received
0927 04:59:47.380858 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.380863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.380921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.380986 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.381622 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.382569 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.384886 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.386065 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.386205 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.386273 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.386909 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.387090 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:59:47.388469 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.379744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41387 (local address 127.20.169.66:58767)
0927 04:59:47.380493 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.380508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.380670 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.381035 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.381042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.381073 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.381459 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.381471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.382708 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.382712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.384695 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.384712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.386433 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.386439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.386644 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.386699 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.387809 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.387995 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":374,"threads_started":1}
I20260927 04:59:47.389667 28334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f43a31bb37488f9b799e5a148fad3b" candidate_uuid: "ab7a27d683cb4d51884bfb576d629c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245a22d06014d6d9f47477d25dcd6fc" is_pre_election: true
I20260927 04:59:47.390233 28334 raft_consensus.cc:2454] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7a27d683cb4d51884bfb576d629c97 in term 0.
I20260927 04:59:47.391012 28420 leader_election.cc:302] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7a27d683cb4d51884bfb576d629c97, f245a22d06014d6d9f47477d25dcd6fc; no voters:
I20260927 04:59:47.391511 28776 raft_consensus.cc:2781] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:47.391639 28776 raft_consensus.cc:482] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:47.391736 28776 raft_consensus.cc:3037] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:47.395769 28776 raft_consensus.cc:504] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.396260 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f43a31bb37488f9b799e5a148fad3b" candidate_uuid: "ab7a27d683cb4d51884bfb576d629c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1769503332dc41ee90ab181cfec04f8f" is_pre_election: true
I20260927 04:59:47.396484 28627 raft_consensus.cc:2454] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7a27d683cb4d51884bfb576d629c97 in term 0.
I20260927 04:59:47.397166 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f43a31bb37488f9b799e5a148fad3b" candidate_uuid: "ab7a27d683cb4d51884bfb576d629c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1769503332dc41ee90ab181cfec04f8f"
I20260927 04:59:47.397372 28627 raft_consensus.cc:3037] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:47.397665 28334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f43a31bb37488f9b799e5a148fad3b" candidate_uuid: "ab7a27d683cb4d51884bfb576d629c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245a22d06014d6d9f47477d25dcd6fc"
I20260927 04:59:47.397861 28334 raft_consensus.cc:3037] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:47.398584 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.386457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:43509 (local address 127.20.169.65:41387)
0927 04:59:47.387759 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.387767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.388322 (+ 555us) server_negotiation.cc:407] Connection header received
0927 04:59:47.388606 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.388612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.388688 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.388755 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.389670 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.391303 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.394752 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.396669 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.396878 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.397002 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.397846 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.398052 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"threads_started":1}
I20260927 04:59:47.399993 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.386102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:41387 (local address 127.20.169.67:43509)
0927 04:59:47.388216 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.388236 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.388445 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.389004 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.389012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.389051 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.389445 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.389462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.391628 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.391638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.394484 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.394508 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.397249 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.397260 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.397574 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.397634 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.398801 (+ 1167us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.399529 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006}
I20260927 04:59:47.400482 28627 raft_consensus.cc:2454] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7a27d683cb4d51884bfb576d629c97 in term 1.
I20260927 04:59:47.400701 28776 leader_election.cc:288] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [CANDIDATE]: Term 1 election: Requested vote from peers 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045), f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387)
I20260927 04:59:47.401003 28420 leader_election.cc:302] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1769503332dc41ee90ab181cfec04f8f, ab7a27d683cb4d51884bfb576d629c97; no voters:
I20260927 04:59:47.400970 28333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f43a31bb37488f9b799e5a148fad3b" candidate_uuid: "1769503332dc41ee90ab181cfec04f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f245a22d06014d6d9f47477d25dcd6fc" is_pre_election: true
I20260927 04:59:47.401340 28776 raft_consensus.cc:2781] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:47.401899 28334 raft_consensus.cc:2454] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7a27d683cb4d51884bfb576d629c97 in term 1.
I20260927 04:59:47.402626 28776 raft_consensus.cc:686] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 1 LEADER]: Becoming Leader. State: Replica: ab7a27d683cb4d51884bfb576d629c97, State: Running, Role: LEADER
I20260927 04:59:47.403081 28776 consensus_queue.cc:237] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } }
I20260927 04:59:47.403757 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.403165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:46133)
0927 04:59:47.403352 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.403598 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 04:59:47.407246 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.386593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:59719 (local address 127.20.169.66:36259)
0927 04:59:47.386787 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.386794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.395744 (+ 8950us) server_negotiation.cc:407] Connection header received
0927 04:59:47.401114 (+ 5370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.401120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.401191 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.401282 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:47.401965 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.403040 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.404966 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.406198 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.406331 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.406396 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.406876 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.407035 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 04:59:47.407934 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.386244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:36259 (local address 127.20.169.67:59719)
0927 04:59:47.395638 (+ 9394us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.395658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:47.400939 (+ 5281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:47.401432 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:47.401437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:47.401461 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:47.401763 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.401777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.403224 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.403228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.404745 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:47.404767 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.406540 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.406546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:47.406711 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.406739 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 04:59:47.407486 (+ 747us) client_negotiation.cc:240] Negotiation successful
0927 04:59:47.407626 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9262,"thread_start_us":9520,"threads_started":1}
I20260927 04:59:47.408560 28485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f43a31bb37488f9b799e5a148fad3b" candidate_uuid: "1769503332dc41ee90ab181cfec04f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7a27d683cb4d51884bfb576d629c97" is_pre_election: true
I20260927 04:59:47.409348 28561 leader_election.cc:302] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1769503332dc41ee90ab181cfec04f8f; no voters: ab7a27d683cb4d51884bfb576d629c97, f245a22d06014d6d9f47477d25dcd6fc
I20260927 04:59:47.409797 28772 raft_consensus.cc:2726] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 04:59:47.409309 28004 catalog_manager.cc:5495] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66). New cstate: current_term: 1 leader_uuid: "ab7a27d683cb4d51884bfb576d629c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7a27d683cb4d51884bfb576d629c97" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 36259 } health_report { overall_health: HEALTHY } } }
I20260927 04:59:47.472780 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.466594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.20.169.66:36259)
0927 04:59:47.466775 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.466781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.466891 (+ 110us) server_negotiation.cc:407] Connection header received
0927 04:59:47.467103 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.467108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.467206 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.467369 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:47.468077 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.468954 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.470386 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.470885 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.471033 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.471470 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:47.472074 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:47.472301 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.472358 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.472448 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 04:59:47.477516 28627 raft_consensus.cc:1261] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Refusing update from remote peer ab7a27d683cb4d51884bfb576d629c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:47.477674 28334 raft_consensus.cc:1261] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Refusing update from remote peer ab7a27d683cb4d51884bfb576d629c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:47.478338 28783 consensus_queue.cc:1035] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1769503332dc41ee90ab181cfec04f8f" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:47.478727 28776 consensus_queue.cc:1035] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f245a22d06014d6d9f47477d25dcd6fc" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:47.481612 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.480909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:58679)
0927 04:59:47.481196 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.481318 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 04:59:47.485466 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.481190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:36195)
0927 04:59:47.484988 (+ 3798us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.485118 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3637}
I20260927 04:59:47.493959 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.493513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.65:37531)
0927 04:59:47.493729 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.493808 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 04:59:47.504089 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.503551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:59813)
0927 04:59:47.503844 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.503939 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260927 04:59:47.512511 28004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:47.512888 28004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:47.517879 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.517395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.66:49505)
0927 04:59:47.517605 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.517695 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 04:59:47.519016 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.517557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.20.169.67:41137)
0927 04:59:47.518735 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.518838 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1071}
I20260927 04:59:47.521176 28004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3543ce954bc14f5f898b6106cbd69262: Sending DeleteTablet for 3 replicas of tablet d8f43a31bb37488f9b799e5a148fad3b
I20260927 04:59:47.522549 28314 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f43a31bb37488f9b799e5a148fad3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:47 UTC) from {username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:51364
I20260927 04:59:47.522552 28465 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f43a31bb37488f9b799e5a148fad3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:47 UTC) from {username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:34844
I20260927 04:59:47.522954 28464 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f43a31bb37488f9b799e5a148fad3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:47 UTC) from {username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:34844
I20260927 04:59:47.522982 28606 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f43a31bb37488f9b799e5a148fad3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:47 UTC) from {username='oryx', principal='oryx/127.20.169.125@KRBTEST.COM'} at 127.0.0.1:39798
I20260927 04:59:47.524227 28798 tablet_replica.cc:330] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: stopping tablet replica
I20260927 04:59:47.524418 28798 raft_consensus.cc:2229] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:47.524652 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 28175
I20260927 04:59:47.524713 28798 raft_consensus.cc:2258] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:47.525357 28797 tablet_replica.cc:330] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: stopping tablet replica
I20260927 04:59:47.525524 28797 raft_consensus.cc:2229] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:47.526055 28797 raft_consensus.cc:2258] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:47.526454 28798 ts_tablet_manager.cc:1905] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:47.527264 28797 ts_tablet_manager.cc:1905] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:47.532843 28798 ts_tablet_manager.cc:1918] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:47.533004 28798 log.cc:1196] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/d8f43a31bb37488f9b799e5a148fad3b
I20260927 04:59:47.533488 28798 ts_tablet_manager.cc:1939] T d8f43a31bb37488f9b799e5a148fad3b P 1769503332dc41ee90ab181cfec04f8f: Deleting consensus metadata
I20260927 04:59:47.533525 28797 ts_tablet_manager.cc:1918] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:47.533730 28797 log.cc:1196] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/d8f43a31bb37488f9b799e5a148fad3b
I20260927 04:59:47.534219 28797 ts_tablet_manager.cc:1939] T d8f43a31bb37488f9b799e5a148fad3b P ab7a27d683cb4d51884bfb576d629c97: Deleting consensus metadata
I20260927 04:59:47.534729 27993 catalog_manager.cc:4841] TS 1769503332dc41ee90ab181cfec04f8f (127.20.169.67:41045): tablet d8f43a31bb37488f9b799e5a148fad3b (table test-table [id=e1be716dee1c4b2bb980eb1b6e6bf29a]) successfully deleted
I20260927 04:59:47.535228 27992 catalog_manager.cc:4841] TS ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259): tablet d8f43a31bb37488f9b799e5a148fad3b (table test-table [id=e1be716dee1c4b2bb980eb1b6e6bf29a]) successfully deleted
I20260927 04:59:47.535635 28796 tablet_replica.cc:330] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: stopping tablet replica
W20260927 04:59:47.535820 27992 catalog_manager.cc:4808] TS ab7a27d683cb4d51884bfb576d629c97 (127.20.169.66:36259): delete failed for tablet d8f43a31bb37488f9b799e5a148fad3b because the tablet was not found. No further retry: Not found: Tablet not found: d8f43a31bb37488f9b799e5a148fad3b
I20260927 04:59:47.535830 28796 raft_consensus.cc:2229] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:47.536132 28796 raft_consensus.cc:2258] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:47.537326 28796 ts_tablet_manager.cc:1905] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:47.543443 28796 ts_tablet_manager.cc:1918] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:47.543588 28796 log.cc:1196] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/d8f43a31bb37488f9b799e5a148fad3b
I20260927 04:59:47.543972 28796 ts_tablet_manager.cc:1939] T d8f43a31bb37488f9b799e5a148fad3b P f245a22d06014d6d9f47477d25dcd6fc: Deleting consensus metadata
I20260927 04:59:47.544888 27993 catalog_manager.cc:4841] TS f245a22d06014d6d9f47477d25dcd6fc (127.20.169.65:41387): tablet d8f43a31bb37488f9b799e5a148fad3b (table test-table [id=e1be716dee1c4b2bb980eb1b6e6bf29a]) successfully deleted
I20260927 04:59:47.556511 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.525677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.20.169.65:41387)
0927 04:59:47.525872 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.525878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.536785 (+ 10907us) server_negotiation.cc:407] Connection header received
0927 04:59:47.537025 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.537031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.537112 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.537225 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:47.539032 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.540270 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.542082 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.542777 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.542925 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.549422 (+ 6497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:47.549441 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:47.549462 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:47.549510 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:47.553538 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:47.554274 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:47.554284 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:47.554287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:47.554463 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:47.554865 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:47.554873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:47.554878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:47.555368 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:47.555658 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.555925 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.556149 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 04:59:47.557348 28375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:47.615623 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.614847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49440)
0927 04:59:47.615293 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.615414 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260927 04:59:47.617197 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.616795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36927 (local address 127.0.0.1:49444)
0927 04:59:47.616978 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:47.617063 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 04:59:47.684782 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 28175
I20260927 04:59:47.709048 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 28392
I20260927 04:59:47.724653 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.709840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.20.169.66:36259)
0927 04:59:47.710083 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.710089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.710111 (+ 22us) server_negotiation.cc:407] Connection header received
0927 04:59:47.710276 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.710283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.710364 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.710473 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:47.711769 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.713032 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.714291 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.715011 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.715135 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.717704 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:47.717720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:47.717750 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:47.717790 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:47.721707 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:47.722441 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:47.722451 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:47.722455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:47.722624 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:47.722998 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:47.723003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:47.723007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:47.723457 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:47.723785 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.724084 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.724333 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 04:59:47.725376 28525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:47.843928 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 28392
I20260927 04:59:47.864234 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 28543
I20260927 04:59:47.880676 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:47.865051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.20.169.67:41045)
0927 04:59:47.865228 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:47.865232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:47.865249 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:47.865409 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:47.865413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:47.865500 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:47.865583 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:47.867205 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.867958 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.869592 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:47.870005 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:47.870147 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:47.873359 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:47.873380 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:47.873402 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:47.873445 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:47.877335 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:47.878160 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:47.878169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:47.878173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:47.878324 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:47.878744 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:47.878749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:47.878752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:47.879196 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:47.879454 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:47.879986 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 04:59:47.880218 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 04:59:47.881435 28667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:47.979904 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 28543
I20260927 04:59:47.998332 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27867
I20260927 04:59:47.999533 28036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:48.122689 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27867
W20260927 04:59:48.144587 28182 connection.cc:535] server connection from 127.0.0.1:38364 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:59:48.144995 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 27970
I20260927 04:59:48.146042 28227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:48.259464 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 27970
2026-09-27T04:59:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10021 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:48 dist-test-slave-39f2 krb5kdc[28810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:48 dist-test-slave-39f2 krb5kdc[28810](info): set up 2 sockets
Sep 27 04:59:48 dist-test-slave-39f2 krb5kdc[28810](info): commencing operation
krb5kdc: starting...
W20260927 04:59:49.401280 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:49 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab.
Entry for principal oryx/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab.
I20260927 04:59:49.620421 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35579
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab
--principal=oryx/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:49.948961 28823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:49.949244 28823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:49.949512 28823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:49.957568 28823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:49.957746 28823 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 04:59:49.957854 28823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:49.957978 28823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:49.958077 28823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:49.958182 28823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:49.968653 28823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.126.keytab
--principal=oryx/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:49.970012 28823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:49.971400 28823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:49.983332 28828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:49.983390 28829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:49.983402 28831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:49.984131 28823 server_base.cc:1034] running on GCE node
I20260927 04:59:49.984930 28823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:49.986275 28823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:49.987604 28823 hybrid_clock.cc:648] HybridClock initialized: now 1790485189987516 us; error 46 us; skew 500 ppm
I20260927 04:59:49.987991 28823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:49 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485189, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:49.991118 28823 init.cc:377] Logged in from keytab as oryx/127.20.169.126@KRBTEST.COM (short username oryx)
I20260927 04:59:49.992597 28823 webserver.cc:460] Webserver started at http://127.20.169.126:38685/ using document root <none> and password file <none>
I20260927 04:59:49.993312 28823 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:49.993402 28823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:49.993664 28823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:49.996510 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "75082f8ce5c44288a2c8890caf07636e"
format_stamp: "Formatted at 2026-09-27 04:59:49 on dist-test-slave-39f2"
server_key: "d6433cb52dd7150418792a67a5969282"
server_key_iv: "bf89ee9b94c0474c55419eff0dab1f5d"
server_key_version: "encryptionkey@0"
I20260927 04:59:49.997324 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "75082f8ce5c44288a2c8890caf07636e"
format_stamp: "Formatted at 2026-09-27 04:59:49 on dist-test-slave-39f2"
server_key: "d6433cb52dd7150418792a67a5969282"
server_key_iv: "bf89ee9b94c0474c55419eff0dab1f5d"
server_key_version: "encryptionkey@0"
I20260927 04:59:50.002254 28823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 04:59:50.005903 28838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:50.007383 28823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 04:59:50.007599 28823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "75082f8ce5c44288a2c8890caf07636e"
format_stamp: "Formatted at 2026-09-27 04:59:49 on dist-test-slave-39f2"
server_key: "d6433cb52dd7150418792a67a5969282"
server_key_iv: "bf89ee9b94c0474c55419eff0dab1f5d"
server_key_version: "encryptionkey@0"
I20260927 04:59:50.007818 28823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:50.023403 28823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:50.027591 28823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:50.027863 28823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:50.049984 28823 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35579
I20260927 04:59:50.049991 28899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35579 every 8 connection(s)
I20260927 04:59:50.051424 28823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:50.055233 28900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:50.061635 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 28823
I20260927 04:59:50.061894 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:50.062563 21157 external_mini_cluster.cc:1428] Setting key fc69169f07fd3f2e3253004d8fbcb8a8
I20260927 04:59:50.066350 28900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Bootstrap starting.
I20260927 04:59:50.070313 28900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:50.071422 28900 log.cc:824] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:50.074571 28900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: No bootstrap required, opened a new log
Sep 27 04:59:50 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:50.083020 28900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082f8ce5c44288a2c8890caf07636e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35579 } }
I20260927 04:59:50.083386 28900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:50.083503 28900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75082f8ce5c44288a2c8890caf07636e, State: Initialized, Role: FOLLOWER
I20260927 04:59:50.084122 28900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082f8ce5c44288a2c8890caf07636e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35579 } }
I20260927 04:59:50.084470 28900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:50.084662 28900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:50.084869 28900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:50.085997 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:50.064628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.20.169.126:35579)
0927 04:59:50.065509 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:50.065522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:50.065566 (+ 44us) server_negotiation.cc:407] Connection header received
0927 04:59:50.067395 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:50.067421 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:50.067848 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:50.068191 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:50.070154 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.071659 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.073249 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.073947 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.074081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:50.078691 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:50.078699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:50.078713 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:50.078757 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:50.081818 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:50.082512 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:50.082518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:50.082525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:50.082699 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:50.083074 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:50.083078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:50.083080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:50.083702 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:50.083992 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:50.084352 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 04:59:50.084872 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":441,"threads_started":1}
I20260927 04:59:50.087337 28900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082f8ce5c44288a2c8890caf07636e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35579 } }
I20260927 04:59:50.087841 28900 leader_election.cc:302] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75082f8ce5c44288a2c8890caf07636e; no voters:
I20260927 04:59:50.088408 28900 leader_election.cc:288] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:50.088557 28905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:50.089267 28905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [term 1 LEADER]: Becoming Leader. State: Replica: 75082f8ce5c44288a2c8890caf07636e, State: Running, Role: LEADER
I20260927 04:59:50.089879 28905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082f8ce5c44288a2c8890caf07636e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35579 } }
I20260927 04:59:50.091037 28900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:50.092478 28908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75082f8ce5c44288a2c8890caf07636e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082f8ce5c44288a2c8890caf07636e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35579 } } }
I20260927 04:59:50.093025 28908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:50.096009 28907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75082f8ce5c44288a2c8890caf07636e. Latest consensus state: current_term: 1 leader_uuid: "75082f8ce5c44288a2c8890caf07636e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75082f8ce5c44288a2c8890caf07636e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35579 } } }
I20260927 04:59:50.096246 28907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:50.097064 28912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:50.101526 28912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:50.111785 28912 catalog_manager.cc:1347] Generated new cluster ID: d7175f772a674f73b2215d02efdbb400
I20260927 04:59:50.111931 28912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:50.138823 28912 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:50.140328 28912 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:50.150382 28912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Generated new TSK 0
I20260927 04:59:50.151523 28912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab.
Entry for principal oryx/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab.
I20260927 04:59:50.222170 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35579
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:50.569094 28927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:50.569392 28927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:50.569499 28927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:50.576783 28927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:50.576941 28927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:50.577040 28927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:59:50.577127 28927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:50.587036 28927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:50.588346 28927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:50.589731 28927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:50.603729 28933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:50.603982 28932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:50.604226 28935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:50.604507 28927 server_base.cc:1034] running on GCE node
I20260927 04:59:50.605170 28927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:50.606068 28927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:50.607285 28927 hybrid_clock.cc:648] HybridClock initialized: now 1790485190607207 us; error 54 us; skew 500 ppm
I20260927 04:59:50.607645 28927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:50 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485190, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:50.610846 28927 init.cc:377] Logged in from keytab as oryx/127.20.169.65@KRBTEST.COM (short username oryx)
I20260927 04:59:50.612419 28927 webserver.cc:460] Webserver started at http://127.20.169.65:40353/ using document root <none> and password file <none>
I20260927 04:59:50.613204 28927 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:50.613332 28927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:50.613655 28927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:50.616547 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "d8b8df1b8e7c4727880b04eefadcc988"
format_stamp: "Formatted at 2026-09-27 04:59:50 on dist-test-slave-39f2"
server_key: "71c3304c639a8d9d79460df88062b02b"
server_key_iv: "28e06d97b7f1b73d1a59c43be8d15778"
server_key_version: "encryptionkey@0"
I20260927 04:59:50.617395 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "d8b8df1b8e7c4727880b04eefadcc988"
format_stamp: "Formatted at 2026-09-27 04:59:50 on dist-test-slave-39f2"
server_key: "71c3304c639a8d9d79460df88062b02b"
server_key_iv: "28e06d97b7f1b73d1a59c43be8d15778"
server_key_version: "encryptionkey@0"
I20260927 04:59:50.622633 28927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 04:59:50.627470 28942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:50.628782 28927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:50.629043 28927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "d8b8df1b8e7c4727880b04eefadcc988"
format_stamp: "Formatted at 2026-09-27 04:59:50 on dist-test-slave-39f2"
server_key: "71c3304c639a8d9d79460df88062b02b"
server_key_iv: "28e06d97b7f1b73d1a59c43be8d15778"
server_key_version: "encryptionkey@0"
I20260927 04:59:50.629253 28927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:50.654949 28927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:50.658962 28927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:50.659221 28927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:50.661921 28927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:50.662022 28927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:50.662223 28927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:50.662308 28927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:59:50 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485190, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:50.687541 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:50.662299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:48845 (local address 127.20.169.126:35579)
0927 04:59:50.662828 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:50.662835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:50.664138 (+ 1303us) server_negotiation.cc:407] Connection header received
0927 04:59:50.666228 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:50.666240 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:50.666344 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:50.666486 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:50.668287 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.669223 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.672710 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.673178 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.673318 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:50.678853 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:50.678860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:50.678867 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:50.678921 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:50.682833 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:50.684055 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:50.684063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:50.684066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:50.684214 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:50.684918 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:50.684925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:50.684930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:50.685406 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:50.685579 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:50.686783 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0927 04:59:50.687057 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260927 04:59:50.688576 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:50.663045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.20.169.65:48845)
0927 04:59:50.663937 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:50.663991 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:50.665932 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:50.666674 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:50.666682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:50.667243 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:50.668107 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:50.668134 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.669416 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.669424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:50.672517 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:50.672541 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.673422 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.673429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:50.673745 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:50.674618 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:50.674655 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:50.678550 (+ 3895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:50.683000 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:50.683014 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:50.683039 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:50.683884 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:50.684369 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:50.684377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:50.684381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:50.684742 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:50.685588 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:50.685602 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:50.686182 (+ 580us) client_negotiation.cc:769] Sending connection context
0927 04:59:50.686639 (+ 457us) client_negotiation.cc:240] Negotiation successful
0927 04:59:50.687345 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":393,"threads_started":1}
I20260927 04:59:50.703518 28927 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:44491
I20260927 04:59:50.703552 29062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:44491 every 8 connection(s)
I20260927 04:59:50.705318 28927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:50.707526 29063 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35579
I20260927 04:59:50.707777 29063 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:50.708600 29063 heartbeater.cc:511] Master 127.20.169.126:35579 requested a full tablet report, sending...
I20260927 04:59:50.709246 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 28927
I20260927 04:59:50.709481 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:50.710237 21157 external_mini_cluster.cc:1428] Setting key 5be91a6649b0a7b7536c27d2aa489a01
I20260927 04:59:50.710994 28855 ts_manager.cc:194] Registered new tserver with Master: d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491)
I20260927 04:59:50.712482 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:50.696082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.20.169.65:60863)
0927 04:59:50.696758 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:50.696785 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:50.696969 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:50.697386 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:50.697391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:50.697910 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:50.698193 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:50.698202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.699834 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.699843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:50.702031 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:50.702052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.703162 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.703171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:50.703415 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:50.704124 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:50.704147 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:50.705232 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:50.709371 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:50.709377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:50.709382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:50.709953 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:50.710395 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:50.710400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:50.710405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:50.710556 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:50.711312 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:50.711316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:50.711522 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 04:59:50.711677 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 04:59:50.712047 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":344,"threads_started":1}
I20260927 04:59:50.713094 28855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:48845
I20260927 04:59:50.717104 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:50.696015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:60863 (local address 127.20.169.126:35579)
0927 04:59:50.696242 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:50.696252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:50.696884 (+ 632us) server_negotiation.cc:407] Connection header received
0927 04:59:50.697115 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:50.697124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:50.697218 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:50.697366 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:50.698342 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.699634 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.702214 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:50.702940 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:50.703093 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:50.705449 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:50.705455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:50.705458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:50.705513 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:50.709223 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:50.710097 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:50.710102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:50.710105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:50.710267 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:50.710685 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:50.710690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:50.710693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:50.711101 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:50.711248 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:50.711808 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 04:59:50.712017 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for oryx/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab.
Entry for principal oryx/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab.
I20260927 04:59:50.757562 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35579
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:51.094461 29067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:51.094758 29067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:51.094861 29067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:51.102306 29067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:51.102422 29067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:51.102556 29067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:59:51.102663 29067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:51.112862 29067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:51.114075 29067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:51.115391 29067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:51.128651 29072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:51.128690 29075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:51.128734 29073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:51.129963 29067 server_base.cc:1034] running on GCE node
I20260927 04:59:51.130661 29067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:51.131618 29067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:51.132931 29067 hybrid_clock.cc:648] HybridClock initialized: now 1790485191132868 us; error 29 us; skew 500 ppm
I20260927 04:59:51.133278 29067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485191, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:51.136530 29067 init.cc:377] Logged in from keytab as oryx/127.20.169.66@KRBTEST.COM (short username oryx)
I20260927 04:59:51.138060 29067 webserver.cc:460] Webserver started at http://127.20.169.66:46633/ using document root <none> and password file <none>
I20260927 04:59:51.138815 29067 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:51.138949 29067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:51.139300 29067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:51.142426 29067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "ccbc57099871417a8d19f87df049b3ee"
format_stamp: "Formatted at 2026-09-27 04:59:51 on dist-test-slave-39f2"
server_key: "eb0d15c41f559f97ca37a6c2d4ea30f9"
server_key_iv: "b9649e2b64febedc0b07696091e8a09c"
server_key_version: "encryptionkey@0"
I20260927 04:59:51.143271 29067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "ccbc57099871417a8d19f87df049b3ee"
format_stamp: "Formatted at 2026-09-27 04:59:51 on dist-test-slave-39f2"
server_key: "eb0d15c41f559f97ca37a6c2d4ea30f9"
server_key_iv: "b9649e2b64febedc0b07696091e8a09c"
server_key_version: "encryptionkey@0"
I20260927 04:59:51.148531 29067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 04:59:51.152449 29082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:51.153872 29067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:59:51.154040 29067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "ccbc57099871417a8d19f87df049b3ee"
format_stamp: "Formatted at 2026-09-27 04:59:51 on dist-test-slave-39f2"
server_key: "eb0d15c41f559f97ca37a6c2d4ea30f9"
server_key_iv: "b9649e2b64febedc0b07696091e8a09c"
server_key_version: "encryptionkey@0"
I20260927 04:59:51.154261 29067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:51.181128 29067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:51.185286 29067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:51.185557 29067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:51.188216 29067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:51.188323 29067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:51.188465 29067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:51.188534 29067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485191, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:51.214730 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.189164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:36339 (local address 127.20.169.126:35579)
0927 04:59:51.189381 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.189386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.190945 (+ 1559us) server_negotiation.cc:407] Connection header received
0927 04:59:51.194147 (+ 3202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.194156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.194275 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.194403 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:51.197143 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.198436 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.200314 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.200959 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.201109 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.205896 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:51.205906 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:51.205914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:51.205967 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:51.210054 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.211091 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.211095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.211097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.211266 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.211904 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.211913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.211918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.212330 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:51.212514 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.214153 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.214365 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 04:59:51.215652 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.189807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.20.169.66:36339)
0927 04:59:51.190773 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.190814 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.193680 (+ 2866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.194684 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.194719 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.195738 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:51.196794 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.196872 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.198614 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.198620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.200144 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.200158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.201274 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.201285 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.201680 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.202600 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:51.202632 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:51.205663 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:51.210215 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.210229 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.210247 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.210862 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.211399 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.211405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.211414 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.211716 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.212655 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:51.212667 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:51.213294 (+ 627us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.213642 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.214289 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":390,"threads_started":1}
I20260927 04:59:51.236150 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.221293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:43971 (local address 127.20.169.126:35579)
0927 04:59:51.221515 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.221523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.222175 (+ 652us) server_negotiation.cc:407] Connection header received
0927 04:59:51.222374 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.222380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.222461 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.222571 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:51.223946 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.224974 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.227001 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.227551 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.227675 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.229946 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:51.229952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:51.229957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:51.230034 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:51.232936 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.233765 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.233772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.233776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.233954 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.234406 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.234415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.234421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.234973 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:51.235167 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.235587 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.235794 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 04:59:51.236232 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.221290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.20.169.66:43971)
0927 04:59:51.222049 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.222078 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.222238 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.222598 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.222603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.223337 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:51.223811 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.223833 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.225112 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.225117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.226849 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.226873 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.227743 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.227749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.227891 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.228606 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:51.228623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:51.229773 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:51.233063 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.233069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.233073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.233644 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.234057 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.234062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.234066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.234251 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.235108 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:51.235112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:51.235264 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.235496 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.235795 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":387,"threads_started":1}
I20260927 04:59:51.241878 29067 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45349
I20260927 04:59:51.241915 29203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45349 every 8 connection(s)
I20260927 04:59:51.243088 29067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:51.245103 29204 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35579
I20260927 04:59:51.245285 29204 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:51.245930 29204 heartbeater.cc:511] Master 127.20.169.126:35579 requested a full tablet report, sending...
I20260927 04:59:51.247309 28855 ts_manager.cc:194] Registered new tserver with Master: ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349)
I20260927 04:59:51.248106 28855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:36339
I20260927 04:59:51.249153 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 29067
I20260927 04:59:51.249341 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:51.249765 21157 external_mini_cluster.cc:1428] Setting key c1273fee357fb5bde01d8ce8fec01ad3
WARNING: no policy specified for oryx/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab.
Entry for principal oryx/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab.
I20260927 04:59:51.296597 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35579
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:51.631973 29208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:51.632287 29208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:51.632464 29208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:51.640094 29208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:51.640244 29208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:51.640385 29208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:59:51.640518 29208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:51.650727 29208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.169.126:35579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:51.652040 29208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:51.653411 29208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:51.666728 29215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:51.666682 29214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:51.666754 29217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:51.667860 29208 server_base.cc:1034] running on GCE node
I20260927 04:59:51.668606 29208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:51.669727 29208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:51.670969 29208 hybrid_clock.cc:648] HybridClock initialized: now 1790485191670890 us; error 43 us; skew 500 ppm
I20260927 04:59:51.671345 29208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485191, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:51.674789 29208 init.cc:377] Logged in from keytab as oryx/127.20.169.67@KRBTEST.COM (short username oryx)
I20260927 04:59:51.676291 29208 webserver.cc:460] Webserver started at http://127.20.169.67:43741/ using document root <none> and password file <none>
I20260927 04:59:51.677083 29208 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:51.677194 29208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:51.677541 29208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:51.680466 29208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "817c7c93eb5245d5ab9754bae95838e2"
format_stamp: "Formatted at 2026-09-27 04:59:51 on dist-test-slave-39f2"
server_key: "a68142ba53490e94aa92d19ed7ec2ce5"
server_key_iv: "004c5fa6f1008767a0d6f2cfc7d6b08c"
server_key_version: "encryptionkey@0"
I20260927 04:59:51.681279 29208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "817c7c93eb5245d5ab9754bae95838e2"
format_stamp: "Formatted at 2026-09-27 04:59:51 on dist-test-slave-39f2"
server_key: "a68142ba53490e94aa92d19ed7ec2ce5"
server_key_iv: "004c5fa6f1008767a0d6f2cfc7d6b08c"
server_key_version: "encryptionkey@0"
I20260927 04:59:51.686280 29208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 04:59:51.689792 29224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:51.691280 29208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:51.691421 29208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "817c7c93eb5245d5ab9754bae95838e2"
format_stamp: "Formatted at 2026-09-27 04:59:51 on dist-test-slave-39f2"
server_key: "a68142ba53490e94aa92d19ed7ec2ce5"
server_key_iv: "004c5fa6f1008767a0d6f2cfc7d6b08c"
server_key_version: "encryptionkey@0"
I20260927 04:59:51.691653 29208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:51.709029 29208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:51.713555 29208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:51.713809 29208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:51.715902 29063 heartbeater.cc:503] Master 127.20.169.126:35579 was elected leader, sending a full tablet report...
I20260927 04:59:51.716938 29208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:51.717049 29208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:51.717252 29208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:51.717391 29208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485191, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:51.744992 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.718360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.20.169.67:58315)
0927 04:59:51.719383 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.719426 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.721556 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.722482 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.722498 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.723411 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:51.724380 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.724418 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.726074 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.726079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.728523 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.728545 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.729755 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.729765 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.730183 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.731214 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:51.731296 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:51.734781 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:51.739067 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.739077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.739093 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.739765 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.740271 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.740277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.740281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.740651 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.741426 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:51.741435 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:51.741873 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.742307 (+ 434us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.743016 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":491,"threads_started":1}
I20260927 04:59:51.745752 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.717550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:58315 (local address 127.20.169.126:35579)
0927 04:59:51.717775 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.717779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.719553 (+ 1774us) server_negotiation.cc:407] Connection header received
0927 04:59:51.721966 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.721975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.722080 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.722201 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:51.724660 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.725887 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.728697 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.729470 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.729622 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.735026 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:51.735036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:51.735042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:51.735095 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:51.738908 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.739913 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.739920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.739924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.740097 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.740860 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.740869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.740875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.741263 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:51.741450 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.745041 (+ 3591us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.745431 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 04:59:51.769969 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.753946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:39139 (local address 127.20.169.126:35579)
0927 04:59:51.754101 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.754105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.754834 (+ 729us) server_negotiation.cc:407] Connection header received
0927 04:59:51.755024 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.755028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.755098 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.755230 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:51.756133 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.756853 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.759540 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.760090 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.760240 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.762497 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:51.762508 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:51.762520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:51.762567 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:51.766844 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.767653 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.767659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.767662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.767842 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.768185 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.768193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.768199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.768616 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:51.768797 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.769360 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.769587 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:59:51.770107 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.754120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.20.169.67:39139)
0927 04:59:51.754712 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.754731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.754885 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.755257 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.755261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.755714 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:51.755995 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.756003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.757018 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.757024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.759344 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.759368 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.760459 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.760465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.760732 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.761453 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:51.761470 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:51.762260 (+ 790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:51.767006 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.767012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.767019 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.767522 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.767956 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:51.767959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:51.767962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:51.768076 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:51.768751 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:51.768756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:51.769107 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.769270 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.769622 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":329,"threads_started":1}
I20260927 04:59:51.770905 29344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:38111 every 8 connection(s)
I20260927 04:59:51.770941 29208 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:38111
I20260927 04:59:51.772702 29208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:51.774994 29345 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35579
I20260927 04:59:51.775250 29345 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:51.775869 29345 heartbeater.cc:511] Master 127.20.169.126:35579 requested a full tablet report, sending...
I20260927 04:59:51.777119 28855 ts_manager.cc:194] Registered new tserver with Master: 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:51.777925 28855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:58315
I20260927 04:59:51.779371 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 29208
I20260927 04:59:51.779575 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:51.779999 21157 external_mini_cluster.cc:1428] Setting key 8cab6890796324be80b8fbb4fdc606cf
I20260927 04:59:51.782768 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485191, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 04:59:51.832687 28950 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:51.822482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.20.169.126:35579)
0927 04:59:51.822699 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.822706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.822878 (+ 172us) server_negotiation.cc:407] Connection header received
0927 04:59:51.823034 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.823038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.823101 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.823186 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:51.824450 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.825224 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.826450 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.826853 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.826945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.832395 (+ 5450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40474: BlockingRecv error: recv got EOF from 127.0.0.1:40474 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 04:59:51 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485191, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:51.848956 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.835657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.20.169.126:35579)
0927 04:59:51.835801 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.835805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.836277 (+ 472us) server_negotiation.cc:407] Connection header received
0927 04:59:51.836423 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.836427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.836498 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.836574 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:51.837713 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.838449 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.839601 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.840082 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.840182 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.843886 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:51.843891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:51.843894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:51.843929 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:51.846503 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.847098 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.847101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.847104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.847218 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:51.847645 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:51.847648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:51.847650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:51.847912 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:51.848150 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.848479 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.848662 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:59:51.853176 28855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:51.860332 28855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:51.884526 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.874399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:38111 (local address 127.0.0.1:38022)
0927 04:59:51.875531 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.875591 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.875804 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.877410 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.877423 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.877473 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:51.878005 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.878023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.879774 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.879782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.881600 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.881629 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.883781 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.883785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.883944 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.883992 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.884104 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.884241 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":3904,"threads_started":1}
I20260927 04:59:51.884765 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.874959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.20.169.67:38111)
0927 04:59:51.876569 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.876581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.876621 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:59:51.876748 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.876759 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.877104 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.878273 (+ 1169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:51.878610 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.879643 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.881846 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.883599 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.883753 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.884041 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.884170 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.884377 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":1184,"threads_started":1}
I20260927 04:59:51.888806 29274 tablet_service.cc:1467] Processing CreateTablet for tablet 0eea716460c14de380250e32fa04a9fb (DEFAULT_TABLE table=test-table [id=376dd414bafd4a89946bbf70d289fb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:51.890455 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eea716460c14de380250e32fa04a9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:51.894718 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.876707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.20.169.65:44491)
0927 04:59:51.886345 (+ 9638us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.886355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.886387 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:59:51.886478 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.886486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.886739 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.886944 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:51.887447 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.888683 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.890893 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.892302 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.892473 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.892682 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.894269 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.894432 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9469,"thread_start_us":9337,"threads_started":1}
I20260927 04:59:51.894757 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.876224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:44491 (local address 127.0.0.1:34432)
0927 04:59:51.885092 (+ 8868us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.885113 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.885207 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.886948 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.886955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.886990 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:51.887242 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.887251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.888885 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.888890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.890709 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.890724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.892475 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.892481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.894117 (+ 1636us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.894139 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.894223 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.894414 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8758,"mutex_wait_us":1,"thread_start_us":11004,"threads_started":1}
I20260927 04:59:51.897466 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.874782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45349 (local address 127.0.0.1:49580)
0927 04:59:51.878645 (+ 3863us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.878666 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.878789 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.888321 (+ 9532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.888326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.888352 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:51.888672 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.888681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.892615 (+ 3934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.892623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.893988 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.894000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.895731 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.895737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.895922 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.895958 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.896917 (+ 959us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.897065 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3719,"mutex_wait_us":21,"thread_start_us":652,"threads_started":1}
I20260927 04:59:51.898093 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.875306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.20.169.66:45349)
0927 04:59:51.887777 (+ 12471us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.887785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.887812 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:59:51.887904 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.887910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.888126 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.888312 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:51.890911 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.892451 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.894140 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.895536 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.895675 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.895981 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.897672 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.897845 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12316,"thread_start_us":12213,"threads_started":1}
I20260927 04:59:51.898787 28993 tablet_service.cc:1467] Processing CreateTablet for tablet 0eea716460c14de380250e32fa04a9fb (DEFAULT_TABLE table=test-table [id=376dd414bafd4a89946bbf70d289fb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:51.900058 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eea716460c14de380250e32fa04a9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:51.901728 29134 tablet_service.cc:1467] Processing CreateTablet for tablet 0eea716460c14de380250e32fa04a9fb (DEFAULT_TABLE table=test-table [id=376dd414bafd4a89946bbf70d289fb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:51.903009 29134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eea716460c14de380250e32fa04a9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:51.909534 29369 tablet_bootstrap.cc:492] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Bootstrap starting.
I20260927 04:59:51.913982 29369 tablet_bootstrap.cc:654] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:51.915583 29369 log.cc:824] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:51.916518 29370 tablet_bootstrap.cc:492] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Bootstrap starting.
I20260927 04:59:51.918403 29372 tablet_bootstrap.cc:492] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Bootstrap starting.
I20260927 04:59:51.920135 29370 tablet_bootstrap.cc:654] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:51.920233 29369 tablet_bootstrap.cc:492] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: No bootstrap required, opened a new log
I20260927 04:59:51.920511 29369 ts_tablet_manager.cc:1397] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 04:59:51.921240 29370 log.cc:824] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:51.921941 29372 tablet_bootstrap.cc:654] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:51.923099 29372 log.cc:824] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:51.924166 29370 tablet_bootstrap.cc:492] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: No bootstrap required, opened a new log
I20260927 04:59:51.924422 29370 ts_tablet_manager.cc:1397] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 04:59:51.931560 29372 tablet_bootstrap.cc:492] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: No bootstrap required, opened a new log
I20260927 04:59:51.931838 29372 ts_tablet_manager.cc:1397] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260927 04:59:51.931661 29370 raft_consensus.cc:348] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.932091 29370 raft_consensus.cc:374] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:51.932211 29370 raft_consensus.cc:729] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b8df1b8e7c4727880b04eefadcc988, State: Initialized, Role: FOLLOWER
I20260927 04:59:51.932909 29370 consensus_queue.cc:260] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.932783 29369 raft_consensus.cc:348] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.933218 29369 raft_consensus.cc:374] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:51.933327 29369 raft_consensus.cc:729] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817c7c93eb5245d5ab9754bae95838e2, State: Initialized, Role: FOLLOWER
I20260927 04:59:51.934019 29369 consensus_queue.cc:260] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.935240 29345 heartbeater.cc:503] Master 127.20.169.126:35579 was elected leader, sending a full tablet report...
I20260927 04:59:51.935787 29369 ts_tablet_manager.cc:1428] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260927 04:59:51.942318 29370 ts_tablet_manager.cc:1428] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260927 04:59:51.943138 29372 raft_consensus.cc:348] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.943588 29372 raft_consensus.cc:374] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:51.943694 29372 raft_consensus.cc:729] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Initialized, Role: FOLLOWER
I20260927 04:59:51.944321 29372 consensus_queue.cc:260] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.945483 29204 heartbeater.cc:503] Master 127.20.169.126:35579 was elected leader, sending a full tablet report...
I20260927 04:59:51.945983 29372 ts_tablet_manager.cc:1428] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260927 04:59:51.958418 29064 tablet.cc:2369] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:51.973320 29375 raft_consensus.cc:482] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:51.973577 29375 raft_consensus.cc:504] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.975293 29375 leader_election.cc:288] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111), ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349)
I20260927 04:59:51.982123 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.975762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:39391 (local address 127.20.169.66:45349)
0927 04:59:51.976008 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.976017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.976868 (+ 851us) server_negotiation.cc:407] Connection header received
0927 04:59:51.977017 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.977020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.977112 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.977208 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:51.977807 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.978509 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.980284 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.981128 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.981253 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.981303 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.981707 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.981851 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 04:59:51.982189 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.975778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:50279 (local address 127.20.169.67:38111)
0927 04:59:51.976023 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:51.976032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:51.976702 (+ 670us) server_negotiation.cc:407] Connection header received
0927 04:59:51.976886 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:51.976892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:51.976977 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:51.977058 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:51.977686 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.978435 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.980358 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.981263 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.981435 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.981510 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:51.981803 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 04:59:51.981961 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 04:59:51.982206 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.975771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45349 (local address 127.20.169.65:39391)
0927 04:59:51.976702 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.976720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.976890 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.977236 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.977239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.977272 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:51.977639 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.977651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.978645 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.978649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.980128 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.980139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.981364 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.981370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.981562 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.981586 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.981661 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.981779 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":23,"thread_start_us":471,"threads_started":1}
I20260927 04:59:51.982321 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:51.975736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:38111 (local address 127.20.169.65:50279)
0927 04:59:51.976597 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:51.976612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:51.976733 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:51.977086 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:51.977090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:51.977115 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:51.977539 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.977549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.978559 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.978565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.980138 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:51.980150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:51.981509 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:51.981513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:51.981623 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:51.981671 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 04:59:51.981762 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:59:51.981881 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":47,"spinlock_wait_cycles":3456,"thread_start_us":529,"threads_started":1}
I20260927 04:59:51.983498 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eea716460c14de380250e32fa04a9fb" candidate_uuid: "d8b8df1b8e7c4727880b04eefadcc988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbc57099871417a8d19f87df049b3ee" is_pre_election: true
I20260927 04:59:51.983498 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eea716460c14de380250e32fa04a9fb" candidate_uuid: "d8b8df1b8e7c4727880b04eefadcc988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2" is_pre_election: true
I20260927 04:59:51.984052 29294 raft_consensus.cc:2454] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b8df1b8e7c4727880b04eefadcc988 in term 0.
I20260927 04:59:51.984053 29159 raft_consensus.cc:2454] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b8df1b8e7c4727880b04eefadcc988 in term 0.
I20260927 04:59:51.984923 28945 leader_election.cc:302] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 817c7c93eb5245d5ab9754bae95838e2, ccbc57099871417a8d19f87df049b3ee, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:51.985376 29375 raft_consensus.cc:2781] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:51.985538 29375 raft_consensus.cc:482] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:51.985646 29375 raft_consensus.cc:3037] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:51.987833 29375 raft_consensus.cc:504] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
I20260927 04:59:51.988479 29375 leader_election.cc:288] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [CANDIDATE]: Term 1 election: Requested vote from peers 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111), ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349)
I20260927 04:59:51.988912 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eea716460c14de380250e32fa04a9fb" candidate_uuid: "d8b8df1b8e7c4727880b04eefadcc988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2"
I20260927 04:59:51.989034 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eea716460c14de380250e32fa04a9fb" candidate_uuid: "d8b8df1b8e7c4727880b04eefadcc988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbc57099871417a8d19f87df049b3ee"
I20260927 04:59:51.989125 29294 raft_consensus.cc:3037] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:51.989265 29159 raft_consensus.cc:3037] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:51.991974 29294 raft_consensus.cc:2454] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b8df1b8e7c4727880b04eefadcc988 in term 1.
I20260927 04:59:51.992018 29159 raft_consensus.cc:2454] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b8df1b8e7c4727880b04eefadcc988 in term 1.
I20260927 04:59:51.992496 28945 leader_election.cc:302] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817c7c93eb5245d5ab9754bae95838e2, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:51.992833 29375 raft_consensus.cc:2781] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:51.993358 29375 raft_consensus.cc:686] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 1 LEADER]: Becoming Leader. State: Replica: d8b8df1b8e7c4727880b04eefadcc988, State: Running, Role: LEADER
I20260927 04:59:51.993857 29375 consensus_queue.cc:237] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } }
W20260927 04:59:51.994937 29205 tablet.cc:2369] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:51.998893 28855 catalog_manager.cc:5495] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65). New cstate: current_term: 1 leader_uuid: "d8b8df1b8e7c4727880b04eefadcc988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } health_report { overall_health: HEALTHY } } }
W20260927 04:59:52.025157 29346 tablet.cc:2369] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:59:52.038789 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.033867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.20.169.65:44491)
0927 04:59:52.034077 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.034083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.034173 (+ 90us) server_negotiation.cc:407] Connection header received
0927 04:59:52.034380 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.034383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.034479 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.034607 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:52.034920 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.035620 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.036894 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.037285 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.037392 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.037772 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.038305 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.038413 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.038523 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.038586 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:59:52.047415 29294 raft_consensus.cc:1261] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Refusing update from remote peer d8b8df1b8e7c4727880b04eefadcc988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.047443 29159 raft_consensus.cc:1261] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Refusing update from remote peer d8b8df1b8e7c4727880b04eefadcc988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.048398 29380 consensus_queue.cc:1035] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.048731 29375 consensus_queue.cc:1035] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.085417 28854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:52.085690 28854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:52.089248 28854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Sending DeleteTablet for 3 replicas of tablet 0eea716460c14de380250e32fa04a9fb
I20260927 04:59:52.090447 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eea716460c14de380250e32fa04a9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:38022
I20260927 04:59:52.090559 29134 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eea716460c14de380250e32fa04a9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 04:59:52.091539 28993 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eea716460c14de380250e32fa04a9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 04:59:52.092063 29392 tablet_replica.cc:330] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: stopping tablet replica
I20260927 04:59:52.092280 29393 tablet_replica.cc:330] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: stopping tablet replica
I20260927 04:59:52.092388 28855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 04:59:52.092495 29392 raft_consensus.cc:2229] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:52.092653 29393 raft_consensus.cc:2229] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 04:59:52.093029 28855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:52.093079 29392 raft_consensus.cc:2258] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.093089 29393 raft_consensus.cc:2258] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.093864 29394 tablet_replica.cc:330] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: stopping tablet replica
I20260927 04:59:52.094244 29394 raft_consensus.cc:2229] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:52.095022 29394 raft_consensus.cc:2258] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.095346 29392 ts_tablet_manager.cc:1905] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.097244 29394 ts_tablet_manager.cc:1905] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.101470 29274 tablet_service.cc:1467] Processing CreateTablet for tablet 9360ea6477274a24bc72105f75f99255 (DEFAULT_TABLE table=test-table [id=c851de7f3d23459d9037d818cb0e2dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:52.101943 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9360ea6477274a24bc72105f75f99255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.102480 29392 ts_tablet_manager.cc:1918] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:52.102634 29392 log.cc:1196] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/0eea716460c14de380250e32fa04a9fb
I20260927 04:59:52.103240 29392 ts_tablet_manager.cc:1939] T 0eea716460c14de380250e32fa04a9fb P 817c7c93eb5245d5ab9754bae95838e2: Deleting consensus metadata
I20260927 04:59:52.104698 29394 ts_tablet_manager.cc:1918] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:52.104882 29394 log.cc:1196] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/0eea716460c14de380250e32fa04a9fb
I20260927 04:59:52.105229 28841 catalog_manager.cc:4841] TS 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111): tablet 0eea716460c14de380250e32fa04a9fb (table test-table [id=376dd414bafd4a89946bbf70d289fb30]) successfully deleted
I20260927 04:59:52.105504 29394 ts_tablet_manager.cc:1939] T 0eea716460c14de380250e32fa04a9fb P d8b8df1b8e7c4727880b04eefadcc988: Deleting consensus metadata
I20260927 04:59:52.106124 29134 tablet_service.cc:1467] Processing CreateTablet for tablet 9360ea6477274a24bc72105f75f99255 (DEFAULT_TABLE table=test-table [id=c851de7f3d23459d9037d818cb0e2dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:52.106154 28993 tablet_service.cc:1467] Processing CreateTablet for tablet 9360ea6477274a24bc72105f75f99255 (DEFAULT_TABLE table=test-table [id=c851de7f3d23459d9037d818cb0e2dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:52.106566 29134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9360ea6477274a24bc72105f75f99255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.106590 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9360ea6477274a24bc72105f75f99255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.108971 29393 ts_tablet_manager.cc:1905] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.111670 28839 catalog_manager.cc:4841] TS d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491): tablet 0eea716460c14de380250e32fa04a9fb (table test-table [id=376dd414bafd4a89946bbf70d289fb30]) successfully deleted
I20260927 04:59:52.114200 29370 tablet_bootstrap.cc:492] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Bootstrap starting.
I20260927 04:59:52.115885 29370 tablet_bootstrap.cc:654] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.115942 29393 ts_tablet_manager.cc:1918] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:52.116076 29393 log.cc:1196] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/0eea716460c14de380250e32fa04a9fb
I20260927 04:59:52.116628 29393 ts_tablet_manager.cc:1939] T 0eea716460c14de380250e32fa04a9fb P ccbc57099871417a8d19f87df049b3ee: Deleting consensus metadata
I20260927 04:59:52.118425 28839 catalog_manager.cc:4841] TS ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349): tablet 0eea716460c14de380250e32fa04a9fb (table test-table [id=376dd414bafd4a89946bbf70d289fb30]) successfully deleted
I20260927 04:59:52.118989 29369 tablet_bootstrap.cc:492] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Bootstrap starting.
I20260927 04:59:52.120270 29372 tablet_bootstrap.cc:492] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Bootstrap starting.
I20260927 04:59:52.120584 29369 tablet_bootstrap.cc:654] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.121933 29372 tablet_bootstrap.cc:654] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.127048 29370 tablet_bootstrap.cc:492] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: No bootstrap required, opened a new log
I20260927 04:59:52.127251 29370 ts_tablet_manager.cc:1397] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260927 04:59:52.127635 29369 tablet_bootstrap.cc:492] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: No bootstrap required, opened a new log
I20260927 04:59:52.127789 29369 ts_tablet_manager.cc:1397] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260927 04:59:52.128134 29370 raft_consensus.cc:348] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.128391 29370 raft_consensus.cc:374] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.128491 29370 raft_consensus.cc:729] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b8df1b8e7c4727880b04eefadcc988, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.128609 29369 raft_consensus.cc:348] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.128877 29369 raft_consensus.cc:374] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.128926 29372 tablet_bootstrap.cc:492] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: No bootstrap required, opened a new log
I20260927 04:59:52.128821 29370 consensus_queue.cc:260] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.128988 29369 raft_consensus.cc:729] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817c7c93eb5245d5ab9754bae95838e2, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.129196 29372 ts_tablet_manager.cc:1397] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260927 04:59:52.129243 29369 consensus_queue.cc:260] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.129796 29369 ts_tablet_manager.cc:1428] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:59:52.130223 29372 raft_consensus.cc:348] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.130514 29372 raft_consensus.cc:374] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.130656 29372 raft_consensus.cc:729] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.131013 29372 consensus_queue.cc:260] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.131693 29372 ts_tablet_manager.cc:1428] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:59:52.132771 29370 ts_tablet_manager.cc:1428] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 04:59:52.195549 29377 raft_consensus.cc:482] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:52.195801 29377 raft_consensus.cc:504] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.197772 29377 leader_election.cc:288] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:52.206525 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.198463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:43813 (local address 127.20.169.67:38111)
0927 04:59:52.198634 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.198639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.199770 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 04:59:52.199993 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.199999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.200088 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.200159 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:52.200952 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.201734 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.204117 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.205395 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.205551 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.205645 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.206113 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.206250 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 04:59:52.206602 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.198195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:42989 (local address 127.20.169.65:44491)
0927 04:59:52.198484 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.198494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.199213 (+ 719us) server_negotiation.cc:407] Connection header received
0927 04:59:52.199580 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.199586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.199720 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.199814 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:52.200407 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.201648 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.204272 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.205546 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.205683 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.205764 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.206175 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.206300 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 04:59:52.207012 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.198202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:38111 (local address 127.20.169.66:43813)
0927 04:59:52.199653 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.199673 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.199810 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.200364 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.200371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.200400 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:52.200746 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.200762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.202190 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.202196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.203959 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.203972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.205622 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.205628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.205901 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.205968 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.206068 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.206205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":317,"thread_start_us":719,"threads_started":1}
I20260927 04:59:52.207386 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.198197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:44491 (local address 127.20.169.66:42989)
0927 04:59:52.199106 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.199126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.199339 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.199860 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.199865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.199896 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:52.200220 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.200233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.201842 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.201848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.204114 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.204129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.205819 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.205825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.205984 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.206007 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.206771 (+ 764us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.206891 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":26,"thread_start_us":620,"threads_started":1}
I20260927 04:59:52.207944 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9360ea6477274a24bc72105f75f99255" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2" is_pre_election: true
I20260927 04:59:52.208228 29294 raft_consensus.cc:2454] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 0.
I20260927 04:59:52.208251 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9360ea6477274a24bc72105f75f99255" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988" is_pre_election: true
I20260927 04:59:52.208760 29013 raft_consensus.cc:2454] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 0.
I20260927 04:59:52.208775 29085 leader_election.cc:302] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817c7c93eb5245d5ab9754bae95838e2, ccbc57099871417a8d19f87df049b3ee; no voters:
I20260927 04:59:52.209226 29377 raft_consensus.cc:2781] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:52.209360 29377 raft_consensus.cc:482] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:52.209465 29377 raft_consensus.cc:3037] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.212677 29377 raft_consensus.cc:504] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.213462 29377 leader_election.cc:288] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 election: Requested vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:52.213953 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9360ea6477274a24bc72105f75f99255" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988"
I20260927 04:59:52.214005 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9360ea6477274a24bc72105f75f99255" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2"
I20260927 04:59:52.214169 29013 raft_consensus.cc:3037] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.214212 29294 raft_consensus.cc:3037] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.216251 29294 raft_consensus.cc:2454] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 1.
I20260927 04:59:52.216639 29085 leader_election.cc:302] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817c7c93eb5245d5ab9754bae95838e2, ccbc57099871417a8d19f87df049b3ee; no voters:
I20260927 04:59:52.216952 29377 raft_consensus.cc:2781] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:52.217478 29377 raft_consensus.cc:686] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 1 LEADER]: Becoming Leader. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Running, Role: LEADER
I20260927 04:59:52.217593 29013 raft_consensus.cc:2454] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 1.
I20260927 04:59:52.217898 29377 consensus_queue.cc:237] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.221661 28854 catalog_manager.cc:5495] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ccbc57099871417a8d19f87df049b3ee (127.20.169.66). New cstate: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } health_report { overall_health: UNKNOWN } } }
Sep 27 04:59:52 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485189, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.126@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:52.285790 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.271287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.20.169.126:35579)
0927 04:59:52.271504 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.271510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.272070 (+ 560us) server_negotiation.cc:407] Connection header received
0927 04:59:52.272220 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.272226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.272310 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.272414 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:52.273571 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.274756 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.275993 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.276646 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.276761 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.280280 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:52.280285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:52.280288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:52.280316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:52.282929 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:52.283777 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:52.283782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:52.283786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:52.283935 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:52.284300 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:52.284303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:52.284305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:52.284549 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:52.284671 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.285171 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.285491 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":79}
I20260927 04:59:52.285894 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.271303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35579 (local address 127.0.0.1:40492)
0927 04:59:52.271957 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.271983 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.272108 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.272460 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.272463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.273190 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:52.273438 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.273447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.274901 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.274905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.275858 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.275869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.276890 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.276898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.277022 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.277694 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:52.277711 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:52.280107 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:52.283048 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:52.283061 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:52.283069 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:52.283660 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:52.284039 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:52.284043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:52.284047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:52.284201 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:52.284691 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:52.284699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:52.284899 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.285086 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.285372 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":354,"threads_started":1}
I20260927 04:59:52.291011 28854 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:40492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 04:59:52.292308 28854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:52.300143 28993 tablet_service.cc:1467] Processing CreateTablet for tablet 64aac05c90ad4dd5867bacb156cd99f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3e45291a67f4da2ac3c6726f7fbc87a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:59:52.300379 29274 tablet_service.cc:1467] Processing CreateTablet for tablet 64aac05c90ad4dd5867bacb156cd99f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3e45291a67f4da2ac3c6726f7fbc87a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:59:52.300925 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aac05c90ad4dd5867bacb156cd99f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.301359 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aac05c90ad4dd5867bacb156cd99f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.305385 29134 tablet_service.cc:1467] Processing CreateTablet for tablet 64aac05c90ad4dd5867bacb156cd99f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3e45291a67f4da2ac3c6726f7fbc87a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:59:52.305996 29134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aac05c90ad4dd5867bacb156cd99f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.308496 29369 tablet_bootstrap.cc:492] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: Bootstrap starting.
I20260927 04:59:52.310012 29369 tablet_bootstrap.cc:654] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.312119 29369 tablet_bootstrap.cc:492] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: No bootstrap required, opened a new log
I20260927 04:59:52.312392 29369 ts_tablet_manager.cc:1397] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260927 04:59:52.313038 29372 tablet_bootstrap.cc:492] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: Bootstrap starting.
I20260927 04:59:52.313198 29369 raft_consensus.cc:348] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.313458 29369 raft_consensus.cc:374] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.313535 29369 raft_consensus.cc:729] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817c7c93eb5245d5ab9754bae95838e2, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.313831 29369 consensus_queue.cc:260] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.314317 29376 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.314618 29376 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.314924 29372 tablet_bootstrap.cc:654] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.317384 29372 tablet_bootstrap.cc:492] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: No bootstrap required, opened a new log
I20260927 04:59:52.317561 29372 ts_tablet_manager.cc:1397] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 04:59:52.317766 29369 ts_tablet_manager.cc:1428] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 04:59:52.318377 29372 raft_consensus.cc:348] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.318640 29372 raft_consensus.cc:374] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.318737 29372 raft_consensus.cc:729] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.319041 29372 consensus_queue.cc:260] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.319504 29377 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.319818 29377 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.320333 29372 ts_tablet_manager.cc:1428] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:59:52.323077 29370 tablet_bootstrap.cc:492] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: Bootstrap starting.
I20260927 04:59:52.325105 29370 tablet_bootstrap.cc:654] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.328047 29370 tablet_bootstrap.cc:492] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: No bootstrap required, opened a new log
I20260927 04:59:52.328253 29370 ts_tablet_manager.cc:1397] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260927 04:59:52.329006 29370 raft_consensus.cc:348] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.329236 29370 raft_consensus.cc:374] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.329353 29370 raft_consensus.cc:729] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b8df1b8e7c4727880b04eefadcc988, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.329646 29370 consensus_queue.cc:260] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.330173 29375 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.330391 29370 ts_tablet_manager.cc:1428] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:59:52.330461 29375 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.481882 29377 raft_consensus.cc:482] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:52.482115 29377 raft_consensus.cc:504] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.482847 29377 leader_election.cc:288] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:52.483544 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aac05c90ad4dd5867bacb156cd99f7" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988" is_pre_election: true
I20260927 04:59:52.483642 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aac05c90ad4dd5867bacb156cd99f7" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2" is_pre_election: true
I20260927 04:59:52.483824 29013 raft_consensus.cc:2454] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 0.
I20260927 04:59:52.483943 29294 raft_consensus.cc:2454] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 0.
I20260927 04:59:52.484270 29083 leader_election.cc:302] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbc57099871417a8d19f87df049b3ee, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:52.484557 29377 raft_consensus.cc:2781] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:52.484685 29377 raft_consensus.cc:482] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:52.484833 29377 raft_consensus.cc:3037] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.486843 29377 raft_consensus.cc:504] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.487571 29377 leader_election.cc:288] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 election: Requested vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:52.488003 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aac05c90ad4dd5867bacb156cd99f7" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988"
I20260927 04:59:52.488013 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aac05c90ad4dd5867bacb156cd99f7" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2"
I20260927 04:59:52.488229 29294 raft_consensus.cc:3037] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.488255 29013 raft_consensus.cc:3037] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.490610 29013 raft_consensus.cc:2454] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 1.
I20260927 04:59:52.491071 29294 raft_consensus.cc:2454] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 1.
I20260927 04:59:52.491070 29083 leader_election.cc:302] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbc57099871417a8d19f87df049b3ee, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:52.491417 29377 raft_consensus.cc:2781] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:52.491644 29377 raft_consensus.cc:686] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [term 1 LEADER]: Becoming Leader. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Running, Role: LEADER
I20260927 04:59:52.491912 29377 consensus_queue.cc:237] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.493774 29400 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: TxnStatusTablet state changed. Reason: New leader ccbc57099871417a8d19f87df049b3ee. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.494081 29400 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:59:52.494719 29412 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 04:59:52.494982 29412 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 04:59:52.495525 28854 catalog_manager.cc:5495] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ccbc57099871417a8d19f87df049b3ee (127.20.169.66). New cstate: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260927 04:59:52.526615 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.517462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45349 (local address 127.0.0.1:47338)
0927 04:59:52.517706 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.517726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.517854 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.521665 (+ 3811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.521672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.521689 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:52.521892 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.521902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.523079 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.523084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.524075 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.524087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.524749 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.524754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.524970 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.525022 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.526027 (+ 1005us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.526048 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.526152 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.526322 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 04:59:52.526661 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.517825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.20.169.66:45349)
0927 04:59:52.521304 (+ 3479us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.521308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.521325 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:52.521399 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.521402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.521477 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.521631 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:52.522064 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.522944 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.524206 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.524607 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.524735 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.525225 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.525922 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.526072 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.526177 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.526254 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3367,"thread_start_us":3196,"threads_started":1}
I20260927 04:59:52.530757 29294 raft_consensus.cc:1261] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Refusing update from remote peer ccbc57099871417a8d19f87df049b3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.531136 29013 raft_consensus.cc:1261] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Refusing update from remote peer ccbc57099871417a8d19f87df049b3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.531710 29377 consensus_queue.cc:1035] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.532459 29377 consensus_queue.cc:1035] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.536084 29376 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: TxnStatusTablet state changed. Reason: New leader ccbc57099871417a8d19f87df049b3ee. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.536435 29376 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.538558 29375 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: TxnStatusTablet state changed. Reason: New leader ccbc57099871417a8d19f87df049b3ee. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.538806 29375 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.541901 29376 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.542130 29376 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P 817c7c93eb5245d5ab9754bae95838e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.542330 29383 mvcc.cc:205] Tried to move back new op lower bound from 7333827348599197696 to 7333827348449550336. Current Snapshot: MvccSnapshot[applied={T|T < 7333827348599197696}]
I20260927 04:59:52.542842 29375 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.542867 29400 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.543097 29375 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P d8b8df1b8e7c4727880b04eefadcc988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:59:52.543098 29400 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:59:52.544039 29400 tablet_replica.cc:439] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } } }
I20260927 04:59:52.544250 29400 tablet_replica.cc:442] T 64aac05c90ad4dd5867bacb156cd99f7 P ccbc57099871417a8d19f87df049b3ee: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:59:52.554939 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.549010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.20.169.66:45349)
0927 04:59:52.549220 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.549225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.549590 (+ 365us) server_negotiation.cc:407] Connection header received
0927 04:59:52.549771 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.549775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.549848 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.549919 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:52.550402 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.551212 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.553209 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.553664 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.553766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.554218 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.554312 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.554402 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.554561 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.554625 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:59:52.571673 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.565207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:48367 (local address 127.20.169.66:45349)
0927 04:59:52.565432 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.565438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.566059 (+ 621us) server_negotiation.cc:407] Connection header received
0927 04:59:52.566206 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.566212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.566317 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.566418 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:52.566921 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.568111 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.569642 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.570358 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.570486 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.570894 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.571022 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.571115 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.571241 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.571314 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:59:52.571698 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.565269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45349 (local address 127.20.169.66:48367)
0927 04:59:52.565959 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.565974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.566088 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.566435 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.566438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.566447 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:59:52.566779 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.566786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.568231 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.568235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.569504 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.569515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.570555 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.570560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.570671 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.570720 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.571120 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.571128 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.571185 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.571281 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":380,"threads_started":1}
I20260927 04:59:52.581324 29294 raft_consensus.cc:1261] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Refusing update from remote peer ccbc57099871417a8d19f87df049b3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.581507 29013 raft_consensus.cc:1261] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Refusing update from remote peer ccbc57099871417a8d19f87df049b3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.581909 29377 consensus_queue.cc:1035] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.582156 29418 consensus_queue.cc:1035] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.588529 29384 mvcc.cc:205] Tried to move back new op lower bound from 7333827348808396800 to 7333827347328471040. Current Snapshot: MvccSnapshot[applied={T|T < 7333827348808396800}]
I20260927 04:59:52.589764 29385 mvcc.cc:205] Tried to move back new op lower bound from 7333827348808396800 to 7333827347328471040. Current Snapshot: MvccSnapshot[applied={T|T < 7333827348808396800}]
I20260927 04:59:52.638301 28854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:52.638602 28854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:52.641810 28854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Sending DeleteTablet for 3 replicas of tablet 9360ea6477274a24bc72105f75f99255
I20260927 04:59:52.642566 28993 tablet_service.cc:1514] Processing DeleteTablet for tablet 9360ea6477274a24bc72105f75f99255 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 04:59:52.642695 29133 tablet_service.cc:1514] Processing DeleteTablet for tablet 9360ea6477274a24bc72105f75f99255 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 04:59:52.642834 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet 9360ea6477274a24bc72105f75f99255 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:38022
I20260927 04:59:52.652220 29427 tablet_replica.cc:330] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: stopping tablet replica
I20260927 04:59:52.652274 29429 tablet_replica.cc:330] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: stopping tablet replica
I20260927 04:59:52.652484 29429 raft_consensus.cc:2229] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:52.652491 29427 raft_consensus.cc:2229] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:52.652714 29427 raft_consensus.cc:2258] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.652714 29429 raft_consensus.cc:2258] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.654297 29427 ts_tablet_manager.cc:1905] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.654660 29429 ts_tablet_manager.cc:1905] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.662199 29427 ts_tablet_manager.cc:1918] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:52.662339 29427 log.cc:1196] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/9360ea6477274a24bc72105f75f99255
I20260927 04:59:52.662775 29427 ts_tablet_manager.cc:1939] T 9360ea6477274a24bc72105f75f99255 P d8b8df1b8e7c4727880b04eefadcc988: Deleting consensus metadata
I20260927 04:59:52.664004 28839 catalog_manager.cc:4841] TS d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491): tablet 9360ea6477274a24bc72105f75f99255 (table test-table [id=c851de7f3d23459d9037d818cb0e2dae]) successfully deleted
I20260927 04:59:52.664676 29429 ts_tablet_manager.cc:1918] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:52.665729 29428 tablet_replica.cc:330] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: stopping tablet replica
I20260927 04:59:52.665942 29428 raft_consensus.cc:2229] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:52.666487 29428 raft_consensus.cc:2258] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.666895 29429 log.cc:1196] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/9360ea6477274a24bc72105f75f99255
I20260927 04:59:52.667340 29429 ts_tablet_manager.cc:1939] T 9360ea6477274a24bc72105f75f99255 P 817c7c93eb5245d5ab9754bae95838e2: Deleting consensus metadata
I20260927 04:59:52.668361 28841 catalog_manager.cc:4841] TS 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111): tablet 9360ea6477274a24bc72105f75f99255 (table test-table [id=c851de7f3d23459d9037d818cb0e2dae]) successfully deleted
I20260927 04:59:52.668751 29428 ts_tablet_manager.cc:1905] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.677367 29428 ts_tablet_manager.cc:1918] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:52.677523 29428 log.cc:1196] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/9360ea6477274a24bc72105f75f99255
I20260927 04:59:52.677935 29428 ts_tablet_manager.cc:1939] T 9360ea6477274a24bc72105f75f99255 P ccbc57099871417a8d19f87df049b3ee: Deleting consensus metadata
I20260927 04:59:52.678933 28839 catalog_manager.cc:4841] TS ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349): tablet 9360ea6477274a24bc72105f75f99255 (table test-table [id=c851de7f3d23459d9037d818cb0e2dae]) successfully deleted
Sep 27 04:59:52 dist-test-slave-39f2 krb5kdc[28810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:59:52 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:59:52 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 04:59:52.715265 28950 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:59:52.700708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.20.169.126:35579)
0927 04:59:52.700941 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.700948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.701332 (+ 384us) server_negotiation.cc:407] Connection header received
0927 04:59:52.701525 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.701531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.701618 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.701717 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:52.702991 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.703803 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.705139 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.705567 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.705696 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.715018 (+ 9322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58064: BlockingRecv error: recv got EOF from 127.0.0.1:58064 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 04:59:52 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM
I20260927 04:59:52.734381 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.718628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.20.169.126:35579)
0927 04:59:52.718775 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.718793 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.721255 (+ 2462us) server_negotiation.cc:407] Connection header received
0927 04:59:52.721425 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.721429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.721503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.721601 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:52.722690 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.723415 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.724632 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.725138 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.725274 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.728896 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:52.728901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:52.728905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:52.728939 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:52.731486 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:52.732168 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:52.732175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:52.732179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:52.732384 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:52.732742 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:52.732747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:52.732750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:52.733175 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:52.733472 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.733791 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.734011 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:59:52.737854 28854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:59:52.738481 28854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:52.745047 29133 tablet_service.cc:1467] Processing CreateTablet for tablet 1530760dc96f4356839960ac220f5c6e (DEFAULT_TABLE table=test-table [id=5bb360bbe598424fa0b158561001cbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:52.745114 28993 tablet_service.cc:1467] Processing CreateTablet for tablet 1530760dc96f4356839960ac220f5c6e (DEFAULT_TABLE table=test-table [id=5bb360bbe598424fa0b158561001cbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:52.745518 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1530760dc96f4356839960ac220f5c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.745518 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1530760dc96f4356839960ac220f5c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.751297 29274 tablet_service.cc:1467] Processing CreateTablet for tablet 1530760dc96f4356839960ac220f5c6e (DEFAULT_TABLE table=test-table [id=5bb360bbe598424fa0b158561001cbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:52.751724 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1530760dc96f4356839960ac220f5c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:52.752708 29372 tablet_bootstrap.cc:492] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Bootstrap starting.
I20260927 04:59:52.754408 29372 tablet_bootstrap.cc:654] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.756539 29372 tablet_bootstrap.cc:492] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: No bootstrap required, opened a new log
I20260927 04:59:52.756691 29372 ts_tablet_manager.cc:1397] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 04:59:52.757478 29372 raft_consensus.cc:348] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.757754 29372 raft_consensus.cc:374] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.757831 29369 tablet_bootstrap.cc:492] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Bootstrap starting.
I20260927 04:59:52.757866 29372 raft_consensus.cc:729] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.758097 29372 consensus_queue.cc:260] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.758700 29372 ts_tablet_manager.cc:1428] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 04:59:52.759276 29369 tablet_bootstrap.cc:654] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.761904 29369 tablet_bootstrap.cc:492] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: No bootstrap required, opened a new log
I20260927 04:59:52.762213 29369 ts_tablet_manager.cc:1397] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 04:59:52.762667 29370 tablet_bootstrap.cc:492] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Bootstrap starting.
I20260927 04:59:52.763090 29369 raft_consensus.cc:348] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.763422 29369 raft_consensus.cc:374] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.763552 29369 raft_consensus.cc:729] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817c7c93eb5245d5ab9754bae95838e2, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.763800 29369 consensus_queue.cc:260] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.764322 29370 tablet_bootstrap.cc:654] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:52.764720 29369 ts_tablet_manager.cc:1428] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 04:59:52.778703 29370 tablet_bootstrap.cc:492] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: No bootstrap required, opened a new log
I20260927 04:59:52.778903 29370 ts_tablet_manager.cc:1397] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260927 04:59:52.779569 29370 raft_consensus.cc:348] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.779796 29370 raft_consensus.cc:374] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:52.779897 29370 raft_consensus.cc:729] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b8df1b8e7c4727880b04eefadcc988, State: Initialized, Role: FOLLOWER
I20260927 04:59:52.780169 29370 consensus_queue.cc:260] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.780656 29370 ts_tablet_manager.cc:1428] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:59:52.898218 29376 raft_consensus.cc:482] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:52.898504 29376 raft_consensus.cc:504] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.899993 29376 leader_election.cc:288] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349)
I20260927 04:59:52.907480 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.900175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:44491 (local address 127.20.169.67:58571)
0927 04:59:52.900699 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.900721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.900910 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.901786 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.901791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.901818 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:52.902149 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.902163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.903601 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.903606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.905279 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.905291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.906616 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.906623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.906798 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.906836 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.906943 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.907084 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":104,"thread_start_us":274,"threads_started":1}
I20260927 04:59:52.908260 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.900345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:58571 (local address 127.20.169.65:44491)
0927 04:59:52.901456 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.901462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.901482 (+ 20us) server_negotiation.cc:407] Connection header received
0927 04:59:52.901556 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.901561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.901638 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.901765 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:52.902332 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.903453 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.905458 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.906408 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.906567 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.906644 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.907380 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.907522 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":880,"threads_started":1}
I20260927 04:59:52.908708 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.901277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:53205 (local address 127.20.169.66:45349)
0927 04:59:52.901479 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.901484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.901815 (+ 331us) server_negotiation.cc:407] Connection header received
0927 04:59:52.902129 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.902136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.902258 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.902342 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:59:52.902922 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.903689 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.906232 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.907493 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.907657 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.907718 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.908291 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.908446 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 04:59:52.908995 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1530760dc96f4356839960ac220f5c6e" candidate_uuid: "817c7c93eb5245d5ab9754bae95838e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988" is_pre_election: true
I20260927 04:59:52.909258 29013 raft_consensus.cc:2454] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817c7c93eb5245d5ab9754bae95838e2 in term 0.
I20260927 04:59:52.909373 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.900884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45349 (local address 127.20.169.67:53205)
0927 04:59:52.901704 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:52.901723 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:52.901938 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:52.902380 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:52.902384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:52.902410 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:59:52.902755 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.902763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.903838 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.903844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.906056 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:52.906072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.907903 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.907910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:52.908086 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.908119 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 04:59:52.908916 (+ 797us) client_negotiation.cc:240] Negotiation successful
0927 04:59:52.909050 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":602,"threads_started":1}
I20260927 04:59:52.909721 29225 leader_election.cc:302] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817c7c93eb5245d5ab9754bae95838e2, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:52.910171 29376 raft_consensus.cc:2781] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:52.910296 29376 raft_consensus.cc:482] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:52.910385 29376 raft_consensus.cc:3037] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.910673 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1530760dc96f4356839960ac220f5c6e" candidate_uuid: "817c7c93eb5245d5ab9754bae95838e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbc57099871417a8d19f87df049b3ee" is_pre_election: true
I20260927 04:59:52.910940 29159 raft_consensus.cc:2454] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817c7c93eb5245d5ab9754bae95838e2 in term 0.
I20260927 04:59:52.912974 29376 raft_consensus.cc:504] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.913550 29376 leader_election.cc:288] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [CANDIDATE]: Term 1 election: Requested vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349)
I20260927 04:59:52.914117 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1530760dc96f4356839960ac220f5c6e" candidate_uuid: "817c7c93eb5245d5ab9754bae95838e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbc57099871417a8d19f87df049b3ee"
I20260927 04:59:52.914098 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1530760dc96f4356839960ac220f5c6e" candidate_uuid: "817c7c93eb5245d5ab9754bae95838e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988"
I20260927 04:59:52.914322 29013 raft_consensus.cc:3037] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.914337 29159 raft_consensus.cc:3037] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:52.916891 29159 raft_consensus.cc:2454] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817c7c93eb5245d5ab9754bae95838e2 in term 1.
I20260927 04:59:52.917341 29225 leader_election.cc:302] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817c7c93eb5245d5ab9754bae95838e2, ccbc57099871417a8d19f87df049b3ee; no voters:
I20260927 04:59:52.917465 29013 raft_consensus.cc:2454] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817c7c93eb5245d5ab9754bae95838e2 in term 1.
I20260927 04:59:52.917690 29376 raft_consensus.cc:2781] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:52.921066 29376 raft_consensus.cc:686] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 1 LEADER]: Becoming Leader. State: Replica: 817c7c93eb5245d5ab9754bae95838e2, State: Running, Role: LEADER
I20260927 04:59:52.921459 29376 consensus_queue.cc:237] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:52.925282 28854 catalog_manager.cc:5495] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "817c7c93eb5245d5ab9754bae95838e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } health_report { overall_health: HEALTHY } } }
I20260927 04:59:52.956768 29452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:52.950901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.20.169.67:38111)
0927 04:59:52.951521 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:52.951529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:52.951553 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:59:52.951652 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:52.951655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:52.951786 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:52.951965 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:52.952410 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.953299 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.954574 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:52.955027 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:52.955152 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:52.955595 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:52.956148 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:52.956296 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:52.956352 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 04:59:52.956427 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":277,"threads_started":1}
I20260927 04:59:52.960639 29013 raft_consensus.cc:1261] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Refusing update from remote peer 817c7c93eb5245d5ab9754bae95838e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.960641 29159 raft_consensus.cc:1261] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Refusing update from remote peer 817c7c93eb5245d5ab9754bae95838e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:52.961359 29449 consensus_queue.cc:1035] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.961728 29376 consensus_queue.cc:1035] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:52.987296 28854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:52.987582 28854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:52.990947 28854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Sending DeleteTablet for 3 replicas of tablet 1530760dc96f4356839960ac220f5c6e
I20260927 04:59:52.991737 28993 tablet_service.cc:1514] Processing DeleteTablet for tablet 1530760dc96f4356839960ac220f5c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 04:59:52.991858 29133 tablet_service.cc:1514] Processing DeleteTablet for tablet 1530760dc96f4356839960ac220f5c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 04:59:52.992012 29427 tablet_replica.cc:330] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: stopping tablet replica
I20260927 04:59:52.992049 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1530760dc96f4356839960ac220f5c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:52 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:38022
I20260927 04:59:52.992137 29428 tablet_replica.cc:330] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: stopping tablet replica
I20260927 04:59:52.992203 29427 raft_consensus.cc:2229] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:52.992342 29428 raft_consensus.cc:2229] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:52.992425 29427 raft_consensus.cc:2258] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.992584 29428 raft_consensus.cc:2258] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:52.993219 29429 tablet_replica.cc:330] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: stopping tablet replica
I20260927 04:59:52.993414 29429 raft_consensus.cc:2229] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:52.993631 28854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 04:59:52.993817 29428 ts_tablet_manager.cc:1905] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:52.993846 29429 raft_consensus.cc:2258] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 04:59:52.994451 28854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:59:52.994807 29429 ts_tablet_manager.cc:1905] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:53.001163 29429 ts_tablet_manager.cc:1918] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:53.001312 29429 log.cc:1196] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/1530760dc96f4356839960ac220f5c6e
I20260927 04:59:53.002053 29428 ts_tablet_manager.cc:1918] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:53.002113 28993 tablet_service.cc:1467] Processing CreateTablet for tablet be450631ccbc42a0a7e0a302136a06a2 (DEFAULT_TABLE table=test-table [id=86efbaf4301e45aea76913a28d0d52be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:53.002144 29133 tablet_service.cc:1467] Processing CreateTablet for tablet be450631ccbc42a0a7e0a302136a06a2 (DEFAULT_TABLE table=test-table [id=86efbaf4301e45aea76913a28d0d52be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:53.002563 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be450631ccbc42a0a7e0a302136a06a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:53.002578 29429 ts_tablet_manager.cc:1939] T 1530760dc96f4356839960ac220f5c6e P 817c7c93eb5245d5ab9754bae95838e2: Deleting consensus metadata
I20260927 04:59:53.002597 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be450631ccbc42a0a7e0a302136a06a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:53.005234 28841 catalog_manager.cc:4841] TS 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111): tablet 1530760dc96f4356839960ac220f5c6e (table test-table [id=5bb360bbe598424fa0b158561001cbc9]) successfully deleted
I20260927 04:59:53.005421 29427 ts_tablet_manager.cc:1905] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:53.009335 29372 tablet_bootstrap.cc:492] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Bootstrap starting.
I20260927 04:59:53.009495 29274 tablet_service.cc:1467] Processing CreateTablet for tablet be450631ccbc42a0a7e0a302136a06a2 (DEFAULT_TABLE table=test-table [id=86efbaf4301e45aea76913a28d0d52be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:59:53.009920 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be450631ccbc42a0a7e0a302136a06a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:53.011170 29372 tablet_bootstrap.cc:654] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:53.012166 29370 tablet_bootstrap.cc:492] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Bootstrap starting.
I20260927 04:59:53.014106 29370 tablet_bootstrap.cc:654] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:53.015197 29428 log.cc:1196] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/1530760dc96f4356839960ac220f5c6e
I20260927 04:59:53.016003 29428 ts_tablet_manager.cc:1939] T 1530760dc96f4356839960ac220f5c6e P ccbc57099871417a8d19f87df049b3ee: Deleting consensus metadata
I20260927 04:59:53.018697 28839 catalog_manager.cc:4841] TS ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349): tablet 1530760dc96f4356839960ac220f5c6e (table test-table [id=5bb360bbe598424fa0b158561001cbc9]) successfully deleted
I20260927 04:59:53.021965 29372 tablet_bootstrap.cc:492] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: No bootstrap required, opened a new log
I20260927 04:59:53.022173 29372 ts_tablet_manager.cc:1397] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260927 04:59:53.022986 29372 raft_consensus.cc:348] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.023371 29372 raft_consensus.cc:374] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:53.023495 29372 raft_consensus.cc:729] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Initialized, Role: FOLLOWER
I20260927 04:59:53.023795 29369 tablet_bootstrap.cc:492] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Bootstrap starting.
I20260927 04:59:53.023851 29372 consensus_queue.cc:260] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.024644 29372 ts_tablet_manager.cc:1428] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 04:59:53.025700 29369 tablet_bootstrap.cc:654] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:53.026859 29370 tablet_bootstrap.cc:492] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: No bootstrap required, opened a new log
I20260927 04:59:53.027091 29370 ts_tablet_manager.cc:1397] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260927 04:59:53.027259 29427 ts_tablet_manager.cc:1918] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:59:53.027675 29427 log.cc:1196] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/1530760dc96f4356839960ac220f5c6e
I20260927 04:59:53.028124 29370 raft_consensus.cc:348] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.028424 29427 ts_tablet_manager.cc:1939] T 1530760dc96f4356839960ac220f5c6e P d8b8df1b8e7c4727880b04eefadcc988: Deleting consensus metadata
I20260927 04:59:53.028528 29370 raft_consensus.cc:374] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:53.028683 29370 raft_consensus.cc:729] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b8df1b8e7c4727880b04eefadcc988, State: Initialized, Role: FOLLOWER
I20260927 04:59:53.029029 29370 consensus_queue.cc:260] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.030475 29370 ts_tablet_manager.cc:1428] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:53.031404 28839 catalog_manager.cc:4841] TS d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491): tablet 1530760dc96f4356839960ac220f5c6e (table test-table [id=5bb360bbe598424fa0b158561001cbc9]) successfully deleted
I20260927 04:59:53.042644 29369 tablet_bootstrap.cc:492] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: No bootstrap required, opened a new log
I20260927 04:59:53.042853 29369 ts_tablet_manager.cc:1397] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20260927 04:59:53.043571 29369 raft_consensus.cc:348] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.043830 29369 raft_consensus.cc:374] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:53.043936 29369 raft_consensus.cc:729] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817c7c93eb5245d5ab9754bae95838e2, State: Initialized, Role: FOLLOWER
I20260927 04:59:53.044212 29369 consensus_queue.cc:260] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.044775 29369 ts_tablet_manager.cc:1428] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 04:59:53.273720 29377 raft_consensus.cc:482] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:59:53.273963 29377 raft_consensus.cc:504] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.274720 29377 leader_election.cc:288] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:53.275622 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be450631ccbc42a0a7e0a302136a06a2" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988" is_pre_election: true
I20260927 04:59:53.275975 29013 raft_consensus.cc:2454] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 0.
I20260927 04:59:53.277159 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be450631ccbc42a0a7e0a302136a06a2" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2" is_pre_election: true
I20260927 04:59:53.277510 29294 raft_consensus.cc:2454] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 0.
I20260927 04:59:53.278455 29083 leader_election.cc:302] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbc57099871417a8d19f87df049b3ee, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:53.279527 29377 raft_consensus.cc:2781] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:59:53.279695 29377 raft_consensus.cc:482] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:59:53.279886 29377 raft_consensus.cc:3037] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:53.283828 29377 raft_consensus.cc:504] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.284978 29013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be450631ccbc42a0a7e0a302136a06a2" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b8df1b8e7c4727880b04eefadcc988"
I20260927 04:59:53.285182 29013 raft_consensus.cc:3037] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:53.288147 29013 raft_consensus.cc:2454] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 1.
I20260927 04:59:53.288643 29083 leader_election.cc:302] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccbc57099871417a8d19f87df049b3ee, d8b8df1b8e7c4727880b04eefadcc988; no voters:
I20260927 04:59:53.292675 29377 leader_election.cc:288] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [CANDIDATE]: Term 1 election: Requested vote from peers d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491), 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111)
I20260927 04:59:53.293192 29377 raft_consensus.cc:2781] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:53.293275 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be450631ccbc42a0a7e0a302136a06a2" candidate_uuid: "ccbc57099871417a8d19f87df049b3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817c7c93eb5245d5ab9754bae95838e2"
I20260927 04:59:53.293540 29294 raft_consensus.cc:3037] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:53.293584 29377 raft_consensus.cc:686] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 1 LEADER]: Becoming Leader. State: Replica: ccbc57099871417a8d19f87df049b3ee, State: Running, Role: LEADER
I20260927 04:59:53.294062 29377 consensus_queue.cc:237] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } }
I20260927 04:59:53.297118 29294 raft_consensus.cc:2454] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbc57099871417a8d19f87df049b3ee in term 1.
I20260927 04:59:53.300953 28853 catalog_manager.cc:5495] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ccbc57099871417a8d19f87df049b3ee (127.20.169.66). New cstate: current_term: 1 leader_uuid: "ccbc57099871417a8d19f87df049b3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbc57099871417a8d19f87df049b3ee" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260927 04:59:53.340761 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:53.335386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.20.169.66:45349)
0927 04:59:53.336197 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:53.336201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:53.336216 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:59:53.336264 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:53.336267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:53.336348 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:53.336454 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:59:53.336960 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.337695 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.339005 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.339499 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.339637 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:53.340045 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:59:53.340147 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:59:53.340240 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:53.340377 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 04:59:53.340463 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699}
I20260927 04:59:53.351811 29013 raft_consensus.cc:1261] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Refusing update from remote peer ccbc57099871417a8d19f87df049b3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:53.352108 29294 raft_consensus.cc:1261] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Refusing update from remote peer ccbc57099871417a8d19f87df049b3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:59:53.352586 29461 consensus_queue.cc:1035] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b8df1b8e7c4727880b04eefadcc988" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:53.353345 29461 consensus_queue.cc:1035] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "817c7c93eb5245d5ab9754bae95838e2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:59:53.430599 28853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:53.430845 28853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:59:53.433753 28853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75082f8ce5c44288a2c8890caf07636e: Sending DeleteTablet for 3 replicas of tablet be450631ccbc42a0a7e0a302136a06a2
I20260927 04:59:53.434525 29134 tablet_service.cc:1514] Processing DeleteTablet for tablet be450631ccbc42a0a7e0a302136a06a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:53 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 04:59:53.434732 28993 tablet_service.cc:1514] Processing DeleteTablet for tablet be450631ccbc42a0a7e0a302136a06a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:53 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 04:59:53.434816 29428 tablet_replica.cc:330] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: stopping tablet replica
I20260927 04:59:53.434991 29428 raft_consensus.cc:2229] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:59:53.435041 29427 tablet_replica.cc:330] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: stopping tablet replica
I20260927 04:59:53.435225 29427 raft_consensus.cc:2229] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:53.435345 29428 raft_consensus.cc:2258] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:53.435434 29427 raft_consensus.cc:2258] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:53.435710 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet be450631ccbc42a0a7e0a302136a06a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:59:53 UTC) from {username='oryx', principal='oryx/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:38022
I20260927 04:59:53.436277 29429 tablet_replica.cc:330] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: stopping tablet replica
I20260927 04:59:53.436470 29429 raft_consensus.cc:2229] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:59:53.436522 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 28927
I20260927 04:59:53.436643 29429 raft_consensus.cc:2258] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:59:53.436673 29428 ts_tablet_manager.cc:1905] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:53.438699 29429 ts_tablet_manager.cc:1905] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:53.443827 29428 ts_tablet_manager.cc:1918] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:53.443956 29428 log.cc:1196] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/be450631ccbc42a0a7e0a302136a06a2
I20260927 04:59:53.444391 29428 ts_tablet_manager.cc:1939] T be450631ccbc42a0a7e0a302136a06a2 P ccbc57099871417a8d19f87df049b3ee: Deleting consensus metadata
I20260927 04:59:53.445780 29427 ts_tablet_manager.cc:1905] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:59:53.453783 29427 ts_tablet_manager.cc:1918] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:53.453987 29427 log.cc:1196] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/be450631ccbc42a0a7e0a302136a06a2
I20260927 04:59:53.454506 29427 ts_tablet_manager.cc:1939] T be450631ccbc42a0a7e0a302136a06a2 P d8b8df1b8e7c4727880b04eefadcc988: Deleting consensus metadata
I20260927 04:59:53.455914 28839 catalog_manager.cc:4841] TS d8b8df1b8e7c4727880b04eefadcc988 (127.20.169.65:44491): tablet be450631ccbc42a0a7e0a302136a06a2 (table test-table [id=86efbaf4301e45aea76913a28d0d52be]) successfully deleted
I20260927 04:59:53.457046 28839 catalog_manager.cc:4841] TS ccbc57099871417a8d19f87df049b3ee (127.20.169.66:45349): tablet be450631ccbc42a0a7e0a302136a06a2 (table test-table [id=86efbaf4301e45aea76913a28d0d52be]) successfully deleted
I20260927 04:59:53.462144 29429 ts_tablet_manager.cc:1918] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:59:53.462353 29429 log.cc:1196] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/be450631ccbc42a0a7e0a302136a06a2
I20260927 04:59:53.462908 29429 ts_tablet_manager.cc:1939] T be450631ccbc42a0a7e0a302136a06a2 P 817c7c93eb5245d5ab9754bae95838e2: Deleting consensus metadata
I20260927 04:59:53.464309 28841 catalog_manager.cc:4841] TS 817c7c93eb5245d5ab9754bae95838e2 (127.20.169.67:38111): tablet be450631ccbc42a0a7e0a302136a06a2 (table test-table [id=86efbaf4301e45aea76913a28d0d52be]) successfully deleted
Sep 27 04:59:53 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.65@KRBTEST.COM
I20260927 04:59:53.479099 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:53.447174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.20.169.65:44491)
0927 04:59:53.464500 (+ 17326us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:53.464507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:53.464994 (+ 487us) server_negotiation.cc:407] Connection header received
0927 04:59:53.465222 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:53.465227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:53.465300 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:53.465406 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:53.466596 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.467302 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.468488 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.469024 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.469156 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:53.472969 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:53.472985 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:53.473007 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:53.473044 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:53.476072 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:53.476870 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:53.476878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:53.476881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:53.477028 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:53.477394 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:53.477400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:53.477404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:53.477841 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:53.478172 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:53.478452 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 04:59:53.478684 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17146,"thread_start_us":17306,"threads_started":1}
I20260927 04:59:53.479780 29058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:53.604434 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 28927
I20260927 04:59:53.627879 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 29067
Sep 27 04:59:53 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.66@KRBTEST.COM
I20260927 04:59:53.643359 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:53.628843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.20.169.66:45349)
0927 04:59:53.629018 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:53.629023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:53.629040 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:59:53.629125 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:53.629129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:53.629191 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:53.629275 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:53.630480 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.631229 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.632427 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.633010 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.633142 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:53.638014 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:53.638033 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:53.638049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:53.638080 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:53.640684 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:53.641428 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:53.641435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:53.641437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:53.641561 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:53.641949 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:53.641952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:53.641954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:53.642285 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:53.642518 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:53.642805 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 04:59:53.642971 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 04:59:53.644102 29199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:53.789894 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 29067
I20260927 04:59:53.817989 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 29208
Sep 27 04:59:53 dist-test-slave-39f2 krb5kdc[28810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.169.67@KRBTEST.COM
I20260927 04:59:53.833341 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:53.818756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.20.169.67:38111)
0927 04:59:53.819276 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:53.819281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:53.819295 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:59:53.819347 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:53.819350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:53.819441 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:53.819558 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:53.820885 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.821600 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.822814 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:53.823211 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:53.823337 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:53.827851 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:53.827868 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:53.827885 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:53.827921 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:53.830641 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:53.831308 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:53.831316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:53.831318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:53.831484 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:53.831832 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:53.831838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:53.831841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:53.832169 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:53.832482 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:53.832761 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 04:59:53.833010 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":263,"threads_started":1}
I20260927 04:59:53.833954 29340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:53.958292 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 29208
I20260927 04:59:53.981223 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 28823
I20260927 04:59:53.982568 28895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:54.106200 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 28823
2026-09-27T04:59:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5845 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:54 dist-test-slave-39f2 krb5kdc[29476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:54 dist-test-slave-39f2 krb5kdc[29476](info): set up 2 sockets
Sep 27 04:59:54 dist-test-slave-39f2 krb5kdc[29476](info): commencing operation
krb5kdc: starting...
W20260927 04:59:55.246068 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:55 dist-test-slave-39f2 krb5kdc[29476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:55.469035 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:37275
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:37275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:55.791162 29489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:55.791447 29489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:55.791628 29489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:55.798627 29489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:55.798720 29489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:55.798765 29489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:55.798815 29489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:55.798887 29489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:55.808477 29489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:37275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:37275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:55.809760 29489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:55.811079 29489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:55.822474 29494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:55.822547 29495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:55.823210 29497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:55.823643 29489 server_base.cc:1034] running on GCE node
I20260927 04:59:55.824380 29489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:55.825765 29489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:55.827037 29489 hybrid_clock.cc:648] HybridClock initialized: now 1790485195826986 us; error 56 us; skew 500 ppm
I20260927 04:59:55.827386 29489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T04:59:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1799 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:59:56 dist-test-slave-39f2 krb5kdc[29503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:59:56 dist-test-slave-39f2 krb5kdc[29503](info): set up 2 sockets
Sep 27 04:59:56 dist-test-slave-39f2 krb5kdc[29503](info): commencing operation
krb5kdc: starting...
W20260927 04:59:57.014104 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 04:59:57 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:59:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 04:59:57.268353 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:42405
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:57.602540 29516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:57.602826 29516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:57.602922 29516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:57.611047 29516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:59:57.611143 29516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:57.611189 29516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:57.611243 29516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:59:57.611294 29516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:59:57.620993 29516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:57.622226 29516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:57.623497 29516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:57.634892 29524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:57.635079 29521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:57.635031 29522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:57.635700 29516 server_base.cc:1034] running on GCE node
I20260927 04:59:57.636466 29516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:57.637917 29516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:57.639143 29516 hybrid_clock.cc:648] HybridClock initialized: now 1790485197639105 us; error 48 us; skew 500 ppm
I20260927 04:59:57.639488 29516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:57 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485197, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:57.642510 29516 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 04:59:57.644001 29516 webserver.cc:460] Webserver started at http://127.20.169.126:34655/ using document root <none> and password file <none>
I20260927 04:59:57.644687 29516 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:57.644877 29516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:57.645192 29516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:57.648289 29516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "d069a39392224c6085c24af2a5a677d3"
format_stamp: "Formatted at 2026-09-27 04:59:57 on dist-test-slave-39f2"
server_key: "bc36b370e4a528d0a9527e1cca270687"
server_key_iv: "ffaf629bca9685c668ff59f49d9039c8"
server_key_version: "encryptionkey@0"
I20260927 04:59:57.649396 29516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "d069a39392224c6085c24af2a5a677d3"
format_stamp: "Formatted at 2026-09-27 04:59:57 on dist-test-slave-39f2"
server_key: "bc36b370e4a528d0a9527e1cca270687"
server_key_iv: "ffaf629bca9685c668ff59f49d9039c8"
server_key_version: "encryptionkey@0"
I20260927 04:59:57.655217 29516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 04:59:57.658743 29531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:57.660161 29516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:59:57.660356 29516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "d069a39392224c6085c24af2a5a677d3"
format_stamp: "Formatted at 2026-09-27 04:59:57 on dist-test-slave-39f2"
server_key: "bc36b370e4a528d0a9527e1cca270687"
server_key_iv: "ffaf629bca9685c668ff59f49d9039c8"
server_key_version: "encryptionkey@0"
I20260927 04:59:57.660552 29516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:57.674520 29516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:57.678795 29516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:57.679260 29516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:57.699800 29516 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:42405
I20260927 04:59:57.699803 29582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:42405 every 8 connection(s)
I20260927 04:59:57.701444 29516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 04:59:57.705569 29583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:59:57.710863 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 29516
I20260927 04:59:57.711123 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 04:59:57.712078 21157 external_mini_cluster.cc:1428] Setting key 961c995ace8f02fa83785436e00d2cad
I20260927 04:59:57.716446 29583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Bootstrap starting.
I20260927 04:59:57.720316 29583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Neither blocks nor log segments found. Creating new log.
I20260927 04:59:57.721408 29583 log.cc:824] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:59:57.724447 29583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: No bootstrap required, opened a new log
Sep 27 04:59:57 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:57.733676 29583 raft_consensus.cc:348] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 04:59:57.734182 29583 raft_consensus.cc:374] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:59:57.734319 29583 raft_consensus.cc:729] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d069a39392224c6085c24af2a5a677d3, State: Initialized, Role: FOLLOWER
I20260927 04:59:57.735138 29583 consensus_queue.cc:260] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 04:59:57.735458 29583 raft_consensus.cc:388] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:59:57.735621 29583 raft_consensus.cc:482] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:59:57.735841 29583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:59:57.738391 29583 raft_consensus.cc:504] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 04:59:57.738901 29583 leader_election.cc:302] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d069a39392224c6085c24af2a5a677d3; no voters:
I20260927 04:59:57.739632 29583 leader_election.cc:288] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:59:57.739802 29588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:59:57.741801 29588 raft_consensus.cc:686] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 LEADER]: Becoming Leader. State: Replica: d069a39392224c6085c24af2a5a677d3, State: Running, Role: LEADER
I20260927 04:59:57.742415 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:57.714381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.20.169.126:42405)
0927 04:59:57.715197 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:57.715209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:57.715251 (+ 42us) server_negotiation.cc:407] Connection header received
0927 04:59:57.717270 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:57.717295 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:57.717677 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:57.718033 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:57.719949 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:57.721502 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:57.723103 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:57.723820 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:57.723958 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:57.733580 (+ 9622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:57.733591 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:57.733613 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:57.733680 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:57.737474 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:57.738207 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:57.738222 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:57.738230 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:57.738416 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:57.738854 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:57.738861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:57.738866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:57.739536 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:57.739856 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:57.740414 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 04:59:57.740939 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":393,"threads_started":1}
I20260927 04:59:57.742357 29588 consensus_queue.cc:237] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 04:59:57.742895 29583 sys_catalog.cc:564] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:59:57.747473 29590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d069a39392224c6085c24af2a5a677d3. Latest consensus state: current_term: 1 leader_uuid: "d069a39392224c6085c24af2a5a677d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } } }
I20260927 04:59:57.747440 29589 sys_catalog.cc:455] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d069a39392224c6085c24af2a5a677d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } } }
I20260927 04:59:57.747939 29590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:57.748020 29589 sys_catalog.cc:458] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: This master's current role is: LEADER
I20260927 04:59:57.748925 29597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:59:57.755231 29597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:59:57.764930 29597 catalog_manager.cc:1347] Generated new cluster ID: d0ec13c7721641b3bad1df0bf2b42165
I20260927 04:59:57.765055 29597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:59:57.814282 29597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:59:57.815721 29597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:59:57.825384 29597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Generated new TSK 0
I20260927 04:59:57.826520 29597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 04:59:57.903924 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42405
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:58.238897 29610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:58.239166 29610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:58.239326 29610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:58.246758 29610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:58.246863 29610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:58.247071 29610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 04:59:58.256892 29610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:58.258337 29610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:58.259629 29610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:58.272535 29618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:58.272686 29616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:58.273145 29615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:58.273638 29610 server_base.cc:1034] running on GCE node
I20260927 04:59:58.274211 29610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:58.275234 29610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:58.276466 29610 hybrid_clock.cc:648] HybridClock initialized: now 1790485198276429 us; error 64 us; skew 500 ppm
I20260927 04:59:58.276801 29610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:58 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:58.279938 29610 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 04:59:58.281515 29610 webserver.cc:460] Webserver started at http://127.20.169.65:39995/ using document root <none> and password file <none>
I20260927 04:59:58.282207 29610 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:58.282354 29610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:58.282665 29610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:58.285526 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "50bfbe63eeb94fe29bd3fb7fef06c9f2"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "97a24e2ec4817df2953e16e49fa4d1d2"
server_key_iv: "c033e0cc6777070b22c6da2ca46ce851"
server_key_version: "encryptionkey@0"
I20260927 04:59:58.286293 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "50bfbe63eeb94fe29bd3fb7fef06c9f2"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "97a24e2ec4817df2953e16e49fa4d1d2"
server_key_iv: "c033e0cc6777070b22c6da2ca46ce851"
server_key_version: "encryptionkey@0"
I20260927 04:59:58.291221 29610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 04:59:58.294533 29625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:58.295930 29610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 04:59:58.296140 29610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "50bfbe63eeb94fe29bd3fb7fef06c9f2"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "97a24e2ec4817df2953e16e49fa4d1d2"
server_key_iv: "c033e0cc6777070b22c6da2ca46ce851"
server_key_version: "encryptionkey@0"
I20260927 04:59:58.296375 29610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:58.314035 29610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:58.318300 29610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:58.318605 29610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:58.319610 29610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:58.321058 29610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:58.321166 29610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:58.321334 29610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:58.321415 29610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:58.356937 29610 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:35747
I20260927 04:59:58.356958 29737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:35747 every 8 connection(s)
I20260927 04:59:58.358368 29610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 04:59:58.359792 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 29610
I20260927 04:59:58.360028 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 04:59:58.360785 21157 external_mini_cluster.cc:1428] Setting key bd886404eeab57d8bf143cceb58efbf8
Sep 27 04:59:58 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:58.378885 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:58.361297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:42767 (local address 127.20.169.126:42405)
0927 04:59:58.361824 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:58.361831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:58.363049 (+ 1218us) server_negotiation.cc:407] Connection header received
0927 04:59:58.365240 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:58.365246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:58.365358 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:58.365472 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:58.366929 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.367626 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.369245 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.369653 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.369793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:58.373139 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:58.373144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:58.373147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:58.373178 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:58.375861 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:58.376590 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:58.376598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:58.376606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:58.376723 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:58.377188 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:58.377192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:58.377194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:58.377462 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:58.377619 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:58.378283 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 04:59:58.378508 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":234,"threads_started":1}
I20260927 04:59:58.379916 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:58.362063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:42767)
0927 04:59:58.362912 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:58.362952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:58.364991 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:58.365705 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:58.365714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:58.366222 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:58.366762 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:58.366778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.367814 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.367825 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:58.369101 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:58.369113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.369891 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.369900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:58.370140 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:58.370799 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:58.370832 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:58.372931 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:58.375994 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:58.376003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:58.376018 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:58.376464 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:58.376883 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:58.376888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:58.376893 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:58.377062 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:58.377596 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:58.377604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:58.377921 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 04:59:58.378180 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 04:59:58.378684 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":311,"threads_started":1}
I20260927 04:59:58.381749 29738 heartbeater.cc:347] Connected to a master server at 127.20.169.126:42405
I20260927 04:59:58.382055 29738 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:58.382812 29738 heartbeater.cc:511] Master 127.20.169.126:42405 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 04:59:58.385018 29548 ts_manager.cc:194] Registered new tserver with Master: 50bfbe63eeb94fe29bd3fb7fef06c9f2 (127.20.169.65:35747)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 04:59:58.386795 29548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:42767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 04:59:58.411934 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42405
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:58.748500 29744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:58.748857 29744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:58.749035 29744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:58.756258 29744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:58.756374 29744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:58.756559 29744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 04:59:58.766532 29744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:58.767815 29744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:58.769207 29744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:58.782595 29750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:58.782635 29753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:58.782658 29751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:58.783896 29744 server_base.cc:1034] running on GCE node
I20260927 04:59:58.784513 29744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:58.785506 29744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:58.786742 29744 hybrid_clock.cc:648] HybridClock initialized: now 1790485198786691 us; error 62 us; skew 500 ppm
I20260927 04:59:58.787047 29744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:58 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:58.790167 29744 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 04:59:58.791522 29744 webserver.cc:460] Webserver started at http://127.20.169.66:41229/ using document root <none> and password file <none>
I20260927 04:59:58.792121 29744 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:58.792212 29744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:58.792464 29744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:58.795415 29744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "e6f91793c8a44318ae46e57d5e0c74a7"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "c0dc45658ae3a5974ab8b756784a9739"
server_key_iv: "f88108817c0850dc58161e902d1d669f"
server_key_version: "encryptionkey@0"
I20260927 04:59:58.796116 29744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "e6f91793c8a44318ae46e57d5e0c74a7"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "c0dc45658ae3a5974ab8b756784a9739"
server_key_iv: "f88108817c0850dc58161e902d1d669f"
server_key_version: "encryptionkey@0"
I20260927 04:59:58.801321 29744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 04:59:58.804785 29760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:58.806270 29744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 04:59:58.806448 29744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "e6f91793c8a44318ae46e57d5e0c74a7"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "c0dc45658ae3a5974ab8b756784a9739"
server_key_iv: "f88108817c0850dc58161e902d1d669f"
server_key_version: "encryptionkey@0"
I20260927 04:59:58.806622 29744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:58.833026 29744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:58.837226 29744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:58.837456 29744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:58.838341 29744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:58.839653 29744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:58.839722 29744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:58.839803 29744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:58.839846 29744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:58.873347 29744 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:32959
I20260927 04:59:58.873482 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:32959 every 8 connection(s)
I20260927 04:59:58.874630 29744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 04:59:58.884388 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 29744
I20260927 04:59:58.884616 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 04:59:58.885224 21157 external_mini_cluster.cc:1428] Setting key eaf66f4fa0c98fbd60929d7c5260bd13
Sep 27 04:59:58 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485198, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:58.896503 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:58.877232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:38907 (local address 127.20.169.126:42405)
0927 04:59:58.877428 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:58.877432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:58.878727 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 04:59:58.880569 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:58.880578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:58.880668 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:58.880761 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:58.882368 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.883083 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.885247 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.885778 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.885932 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:58.889394 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:58.889404 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:58.889412 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:58.889462 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:58.893224 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:58.893947 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:58.893956 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:58.893963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:58.894137 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:58.894564 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:58.894572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:58.894578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:58.894993 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:58.895159 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:58.895858 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 04:59:58.896096 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 04:59:58.897518 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:58.877759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:38907)
0927 04:59:58.878562 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:58.878607 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:58.880260 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:58.881062 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:58.881071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:58.881609 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:58.882202 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:58.882220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.883250 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.883255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:58.885067 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:58.885093 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:58.886014 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:58.886021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:58.886244 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:58.886938 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:58.886959 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:58.889155 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:58.893371 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:58.893377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:58.893390 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:58.893789 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:58.894249 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:58.894252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:58.894254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:58.894444 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:58.895144 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:58.895150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:58.895455 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 04:59:58.895736 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 04:59:58.896282 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":350,"threads_started":1}
I20260927 04:59:58.899267 29873 heartbeater.cc:347] Connected to a master server at 127.20.169.126:42405
I20260927 04:59:58.899623 29873 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:58.900386 29873 heartbeater.cc:511] Master 127.20.169.126:42405 requested a full tablet report, sending...
I20260927 04:59:58.902030 29548 ts_manager.cc:194] Registered new tserver with Master: e6f91793c8a44318ae46e57d5e0c74a7 (127.20.169.66:32959)
I20260927 04:59:58.903424 29548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:38907
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 04:59:58.940001 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42405
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:59:59.263074 29878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:59:59.263391 29878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:59:59.263490 29878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:59:59.270663 29878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:59:59.270759 29878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:59:59.270874 29878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 04:59:59.279960 29878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 04:59:59.281149 29878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:59:59.282361 29878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:59:59.295074 29883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:59.295164 29884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:59:59.295365 29886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:59:59.296382 29878 server_base.cc:1034] running on GCE node
I20260927 04:59:59.297065 29878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:59:59.297972 29878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:59:59.299225 29878 hybrid_clock.cc:648] HybridClock initialized: now 1790485199299160 us; error 56 us; skew 500 ppm
I20260927 04:59:59.299535 29878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 04:59:59 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485199, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:59:59.302814 29878 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 04:59:59.304091 29878 webserver.cc:460] Webserver started at http://127.20.169.67:40319/ using document root <none> and password file <none>
I20260927 04:59:59.304736 29878 fs_manager.cc:359] Metadata directory not provided
I20260927 04:59:59.304916 29878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:59:59.305243 29878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:59:59.308009 29878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "90465a43a08b4765b4cb812b26dd277f"
format_stamp: "Formatted at 2026-09-27 04:59:59 on dist-test-slave-39f2"
server_key: "41aa84bc819574441580ec523d0439a9"
server_key_iv: "8c6accc77e00b503ebe47c00571352a8"
server_key_version: "encryptionkey@0"
I20260927 04:59:59.308779 29878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "90465a43a08b4765b4cb812b26dd277f"
format_stamp: "Formatted at 2026-09-27 04:59:59 on dist-test-slave-39f2"
server_key: "41aa84bc819574441580ec523d0439a9"
server_key_iv: "8c6accc77e00b503ebe47c00571352a8"
server_key_version: "encryptionkey@0"
I20260927 04:59:59.313920 29878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 04:59:59.317185 29893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:59.318421 29878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 04:59:59.318593 29878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "90465a43a08b4765b4cb812b26dd277f"
format_stamp: "Formatted at 2026-09-27 04:59:59 on dist-test-slave-39f2"
server_key: "41aa84bc819574441580ec523d0439a9"
server_key_iv: "8c6accc77e00b503ebe47c00571352a8"
server_key_version: "encryptionkey@0"
I20260927 04:59:59.318819 29878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:59:59.344420 29878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:59:59.349416 29878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:59:59.349728 29878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:59:59.350816 29878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:59:59.352344 29878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:59:59.352423 29878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:59.352587 29878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:59:59.352646 29878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:59:59.388209 29878 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:40777
I20260927 04:59:59.388249 30007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:40777 every 8 connection(s)
I20260927 04:59:59.389623 29878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 04:59:59.390404 29738 heartbeater.cc:503] Master 127.20.169.126:42405 was elected leader, sending a full tablet report...
I20260927 04:59:59.391732 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 29878
I20260927 04:59:59.391963 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 04:59:59.392571 21157 external_mini_cluster.cc:1428] Setting key 6b80ae96abbf5e6e3faac678172e1383
Sep 27 04:59:59 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485199, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 04:59:59.411247 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:59.392477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:43905 (local address 127.20.169.126:42405)
0927 04:59:59.392662 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:59.392670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:59.394623 (+ 1953us) server_negotiation.cc:407] Connection header received
0927 04:59:59.396610 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:59.396614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:59.396676 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:59.396755 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:59.398563 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.399393 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.401130 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.401550 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.401647 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:59.405087 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:59.405091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:59.405094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:59.405124 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:59.408348 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.409100 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.409109 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.409112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.409230 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.409645 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.409648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.409650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.409958 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:59.410082 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:59.410790 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 04:59:59.411008 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 04:59:59.412191 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:59.393182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:43905)
0927 04:59:59.394455 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0927 04:59:59.394517 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 04:59:59.396363 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:59:59.397026 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:59:59.397034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:59:59.397662 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:59:59.398388 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:59.398409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.399551 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.399555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:59.400969 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:59:59.400982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.401744 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.401749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:59:59.401949 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:59.402645 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:59:59.402669 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:59:59.404873 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:59:59.408490 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:59.408496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:59.408517 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:59.408995 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:59.409371 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:59:59.409374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:59:59.409377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:59:59.409544 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:59:59.410155 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:59:59.410160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:59:59.410451 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 04:59:59.410668 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 04:59:59.411085 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":354,"threads_started":1}
I20260927 04:59:59.413486 30008 heartbeater.cc:347] Connected to a master server at 127.20.169.126:42405
I20260927 04:59:59.413746 30008 heartbeater.cc:464] Registering TS with master...
I20260927 04:59:59.414384 30008 heartbeater.cc:511] Master 127.20.169.126:42405 requested a full tablet report, sending...
I20260927 04:59:59.415799 29548 ts_manager.cc:194] Registered new tserver with Master: 90465a43a08b4765b4cb812b26dd277f (127.20.169.67:40777)
I20260927 04:59:59.416620 29548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:43905
I20260927 04:59:59.420120 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab.
Entry for principal oryx/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab.
WARNING: no policy specified for oryx/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab.
Entry for principal oryx/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab.
WARNING: no policy specified for oryx/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab.
Entry for principal oryx/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab.
I20260927 04:59:59.555137 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 29610
Sep 27 04:59:59 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 04:59:59.571907 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:59.556328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.20.169.65:35747)
0927 04:59:59.557035 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:59.557041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:59.557069 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:59:59.557129 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:59.557135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:59.557352 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:59.557485 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:59.558935 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.559959 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.561394 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.561978 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.562077 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:59.566097 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:59.566109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:59.566123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:59.566154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:59.569302 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.569957 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.569964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.569966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.570099 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.570423 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.570427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.570428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.570762 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:59.570977 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:59.571428 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 04:59:59.571626 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":336,"threads_started":1}
I20260927 04:59:59.573318 29733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:59.659044 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 29610
I20260927 04:59:59.676690 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 29744
Sep 27 04:59:59 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 04:59:59.694034 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:59.677755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.20.169.66:32959)
0927 04:59:59.678417 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:59.678424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:59.678451 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:59:59.678522 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:59.678529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:59.678734 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:59.678877 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:59.680119 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.681319 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.682614 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.683249 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.683384 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:59.687698 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:59.687710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:59.687725 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:59.687761 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:59.691268 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.691895 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.691903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.691905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.692035 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.692428 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.692432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.692434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.692805 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:59.693138 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:59.693456 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 04:59:59.693681 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":341,"threads_started":1}
I20260927 04:59:59.695422 29868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:59.780511 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 29744
I20260927 04:59:59.797466 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 29878
Sep 27 04:59:59 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 04:59:59.814968 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:59:59.798567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.20.169.67:40777)
0927 04:59:59.799191 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 04:59:59.799198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:59:59.799226 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:59:59.799286 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:59:59.799292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:59:59.799562 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:59:59.799735 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:59:59.801191 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.802279 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.803560 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:59:59.804132 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:59:59.804284 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:59:59.808778 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:59:59.808790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:59:59.808845 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:59:59.808888 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:59:59.811610 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.812648 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.812656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.812659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.812859 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:59:59.813354 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:59:59.813358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:59:59.813361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:59:59.813746 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:59:59.814006 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 04:59:59.814350 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 04:59:59.814565 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":328,"threads_started":1}
I20260927 04:59:59.816340 30003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 04:59:59.905762 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 29878
I20260927 04:59:59.922821 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 29516
I20260927 04:59:59.924018 29578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:00.025856 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 29516
I20260927 05:00:00.046932 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:42405
--webserver_interface=127.20.169.126
--webserver_port=34655
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:00:00.367957 30025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:00.368239 30025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:00.368391 30025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:00.375541 30025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:00:00.375643 30025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:00.375728 30025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:00.375833 30025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:00:00.375916 30025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:00:00.385735 30025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=34655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:00.387116 30025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:00.388489 30025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:00.400568 30031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:00.400645 30030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:00.400677 30033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:00.401751 30025 server_base.cc:1034] running on GCE node
I20260927 05:00:00.402506 30025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:00.403921 30025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:00.405118 30025 hybrid_clock.cc:648] HybridClock initialized: now 1790485200405054 us; error 54 us; skew 500 ppm
I20260927 05:00:00.405489 30025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:00.408572 30025 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:00:00.410202 30025 webserver.cc:460] Webserver started at http://127.20.169.126:34655/ using document root <none> and password file <none>
I20260927 05:00:00.410871 30025 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:00.411029 30025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:00.416105 30025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:00:00.418851 30040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:00.420130 30025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:00:00.420334 30025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "d069a39392224c6085c24af2a5a677d3"
format_stamp: "Formatted at 2026-09-27 04:59:57 on dist-test-slave-39f2"
server_key: "bc36b370e4a528d0a9527e1cca270687"
server_key_iv: "ffaf629bca9685c668ff59f49d9039c8"
server_key_version: "encryptionkey@0"
I20260927 05:00:00.421345 30025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:00.439726 30025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:00.443430 30025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:00.443699 30025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:00.462240 30025 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:42405
I20260927 05:00:00.462260 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:42405 every 8 connection(s)
I20260927 05:00:00.463693 30025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:00:00.468124 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30025
I20260927 05:00:00.468969 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:35747
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:42405
--webserver_port=39995
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:00:00.470674 30093 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:00:00.473963 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Bootstrap starting.
I20260927 05:00:00.492694 30093 log.cc:824] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:00.501456 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:00:00.501976 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Bootstrap complete.
I20260927 05:00:00.510949 30093 raft_consensus.cc:348] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 05:00:00.511790 30093 raft_consensus.cc:729] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d069a39392224c6085c24af2a5a677d3, State: Initialized, Role: FOLLOWER
I20260927 05:00:00.512363 30093 consensus_queue.cc:260] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 05:00:00.512606 30093 raft_consensus.cc:388] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:00:00.512735 30093 raft_consensus.cc:482] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:00:00.512941 30093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:00:00.516588 30093 raft_consensus.cc:504] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 05:00:00.517139 30093 leader_election.cc:302] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d069a39392224c6085c24af2a5a677d3; no voters:
I20260927 05:00:00.517777 30093 leader_election.cc:288] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:00:00.517920 30097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:00:00.519428 30097 raft_consensus.cc:686] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [term 2 LEADER]: Becoming Leader. State: Replica: d069a39392224c6085c24af2a5a677d3, State: Running, Role: LEADER
I20260927 05:00:00.520081 30097 consensus_queue.cc:237] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } }
I20260927 05:00:00.520279 30093 sys_catalog.cc:564] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:00:00.522528 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d069a39392224c6085c24af2a5a677d3. Latest consensus state: current_term: 2 leader_uuid: "d069a39392224c6085c24af2a5a677d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } } }
I20260927 05:00:00.522879 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:00.523348 30098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d069a39392224c6085c24af2a5a677d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d069a39392224c6085c24af2a5a677d3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 42405 } } }
I20260927 05:00:00.523653 30098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:00.529250 30102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:00:00.535300 30102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:00:00.536629 30102 catalog_manager.cc:1259] Loaded cluster ID: d0ec13c7721641b3bad1df0bf2b42165
I20260927 05:00:00.536764 30102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:00:00.540771 30102 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:00:00.542407 30102 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d069a39392224c6085c24af2a5a677d3: Loaded TSK: 0
I20260927 05:00:00.543906 30102 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:00:00.806190 30095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:00.806504 30095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:00.806674 30095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:00.813954 30095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:00.814100 30095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:00.814272 30095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:00:00.824092 30095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.65.keytab
--principal=oryx/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:35747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=39995
--tserver_master_addrs=127.20.169.126:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:00.825613 30095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:00.826946 30095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:00.841725 30119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:00.842165 30122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:00.842247 30120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:00.842758 30095 server_base.cc:1034] running on GCE node
I20260927 05:00:00.843425 30095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:00.844481 30095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:00.845743 30095 hybrid_clock.cc:648] HybridClock initialized: now 1790485200845674 us; error 55 us; skew 500 ppm
I20260927 05:00:00.846094 30095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485200, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:00.850140 30095 init.cc:377] Logged in from keytab as oryx/127.20.169.65@KRBTEST.COM (short username oryx)
I20260927 05:00:00.851902 30095 webserver.cc:460] Webserver started at http://127.20.169.65:39995/ using document root <none> and password file <none>
I20260927 05:00:00.852726 30095 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:00.852900 30095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:00.859815 30095 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 05:00:00.863108 30129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:00.864766 30095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:00:00.864998 30095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "50bfbe63eeb94fe29bd3fb7fef06c9f2"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "97a24e2ec4817df2953e16e49fa4d1d2"
server_key_iv: "c033e0cc6777070b22c6da2ca46ce851"
server_key_version: "encryptionkey@0"
I20260927 05:00:00.866320 30095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:00.895048 30095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:00.900043 30095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:00.900254 30095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:00.901240 30095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:00.902510 30095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:00.902591 30095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:00.902671 30095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:00.902724 30095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:00.936154 30095 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:35747
I20260927 05:00:00.936292 30242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:35747 every 8 connection(s)
I20260927 05:00:00.937649 30095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:00:00.941789 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30095
I20260927 05:00:00.942487 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:32959
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:42405
--webserver_port=41229
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:00:00.960263 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:00.940890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:55757 (local address 127.20.169.126:42405)
0927 05:00:00.945303 (+ 4413us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:00.945320 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:00.945375 (+ 55us) server_negotiation.cc:407] Connection header received
0927 05:00:00.946047 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:00.946069 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:00.946520 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:00.946930 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:00.948724 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.950184 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.952136 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.952925 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.953154 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:00.959275 (+ 6121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:55757: BlockingRecv error: recv got EOF from 127.20.169.65:55757 (error 108)
Metrics: {"server-negotiator.queue_time_us":4148,"thread_start_us":3962,"threads_started":1}
I20260927 05:00:00.960371 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:00.940954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:55757)
0927 05:00:00.942009 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:00.942088 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:00.945432 (+ 3344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:00.947317 (+ 1885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:00.947331 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:00.947905 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:00.948514 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:00.948534 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.950369 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.950377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:00.951939 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:00.951953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.953196 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.953204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:00.953500 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:00.954236 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:00.954266 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:00.958953 (+ 4687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":415,"threads_started":1}
W20260927 05:00:00.961158 30243 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:42405 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.169.126:42405: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:00.971975 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:00.961746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:45353)
0927 05:00:00.961964 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:00.961988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:00.962113 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:00.962463 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:00.962470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:00.962940 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:00.963167 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:00.963182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.964317 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.964326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:00.966114 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:00.966146 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.966960 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.966966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:00.967207 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:00.967913 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:00.967938 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:00.971688 (+ 3750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 05:00:00.972456 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:00.961890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:45353 (local address 127.20.169.126:42405)
0927 05:00:00.962046 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:00.962050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:00.962066 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:00:00.962246 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:00.962250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:00.962320 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:00.962394 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:00.963360 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.964153 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.966302 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.966781 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.966914 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:00.971932 (+ 5018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:45353: BlockingRecv error: recv got EOF from 127.20.169.65:45353 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:00 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:00.981997 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:00.972850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:38401)
0927 05:00:00.973023 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:00.973041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:00.973136 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:00.973447 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:00.973452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:00.973820 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:00.973998 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:00.974008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.975112 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.975120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:00.976631 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:00.976645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.977538 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.977544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:00.977730 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:00.978323 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:00.978345 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:00.981711 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:00:00.982179 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:00.972977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:38401 (local address 127.20.169.126:42405)
0927 05:00:00.973144 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:00.973148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:00.973168 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:00:00.973223 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:00.973227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:00.973311 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:00.973410 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:00.974157 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.974957 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.976793 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:00.977386 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:00.977495 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:00.981907 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:38401: BlockingRecv error: recv got EOF from 127.20.169.65:38401 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 05:00:00.982493 30243 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:00:01.325914 30247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:01.326216 30247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:01.326385 30247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:01.333774 30247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:01.333916 30247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:01.334213 30247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:00:01.344209 30247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.66.keytab
--principal=oryx/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:32959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=41229
--tserver_master_addrs=127.20.169.126:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:01.345635 30247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:01.346962 30247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:01.360773 30253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:01.361096 30255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:01.360776 30252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:01.361893 30247 server_base.cc:1034] running on GCE node
I20260927 05:00:01.362612 30247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:01.363574 30247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:01.364854 30247 hybrid_clock.cc:648] HybridClock initialized: now 1790485201364708 us; error 118 us; skew 500 ppm
I20260927 05:00:01.365173 30247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485201, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:01.368487 30247 init.cc:377] Logged in from keytab as oryx/127.20.169.66@KRBTEST.COM (short username oryx)
I20260927 05:00:01.370072 30247 webserver.cc:460] Webserver started at http://127.20.169.66:41229/ using document root <none> and password file <none>
I20260927 05:00:01.370755 30247 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:01.370916 30247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:01.376645 30247 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 05:00:01.379616 30262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:01.381119 30247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:00:01.381291 30247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "e6f91793c8a44318ae46e57d5e0c74a7"
format_stamp: "Formatted at 2026-09-27 04:59:58 on dist-test-slave-39f2"
server_key: "c0dc45658ae3a5974ab8b756784a9739"
server_key_iv: "f88108817c0850dc58161e902d1d669f"
server_key_version: "encryptionkey@0"
I20260927 05:00:01.382233 30247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:01.408466 30247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:01.412935 30247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:01.413197 30247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:01.414249 30247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:01.415640 30247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:01.415876 30247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:01.416018 30247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:01.416106 30247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:01.450804 30247 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:32959
I20260927 05:00:01.450891 30375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:32959 every 8 connection(s)
I20260927 05:00:01.452298 30247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:00:01.458779 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30247
I20260927 05:00:01.459494 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:40777
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:42405
--webserver_port=40319
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:00:01.471573 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.454918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:32933 (local address 127.20.169.126:42405)
0927 05:00:01.455132 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.455138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.456279 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 05:00:01.458116 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.458121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.458189 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.458279 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.460503 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.461355 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.463931 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.464757 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.464920 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.471271 (+ 6351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:32933: BlockingRecv error: recv got EOF from 127.20.169.66:32933 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:00:01.472309 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.455431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:32933)
0927 05:00:01.456109 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.456153 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.457847 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.458602 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.458616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.459406 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.460297 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.460327 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.461580 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.461588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.463736 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.463754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.465100 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.465110 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.465410 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.466224 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.466254 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.471046 (+ 4792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":266,"threads_started":1}
W20260927 05:00:01.473088 30376 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:42405 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.169.126:42405: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:01.484180 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.473705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:34015)
0927 05:00:01.473964 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.473983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.474098 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.474517 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.474521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.474868 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.475032 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.475041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.476557 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.476562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.477663 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.477677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.478791 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.478797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.479069 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.479819 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.479846 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.483781 (+ 3935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144}
W20260927 05:00:01.484193 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.473823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:34015 (local address 127.20.169.126:42405)
0927 05:00:01.474038 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.474045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.474068 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:00:01.474228 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.474236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.474321 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.474450 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.475185 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.476368 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.477819 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.478594 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.478748 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.484010 (+ 5262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:34015: BlockingRecv error: recv got EOF from 127.20.169.66:34015 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:01.494158 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.485045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:41837)
0927 05:00:01.485296 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.485321 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.485442 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.485759 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.485765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.486259 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.486490 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.486505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.487546 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.487552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.489122 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.489136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.489815 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.489822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.490005 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.490604 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.490627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.493833 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 05:00:01.494249 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.485190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:41837 (local address 127.20.169.126:42405)
0927 05:00:01.485343 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.485398 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.485417 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:00:01.485581 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.485585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.485645 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.485719 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.486642 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.487406 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.489255 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.489678 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.489775 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.494033 (+ 4258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:41837: BlockingRecv error: recv got EOF from 127.20.169.66:41837 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 05:00:01.494496 30376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:00:01.798410 30379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:01.798691 30379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:01.798784 30379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:01.805982 30379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:01.806097 30379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:01.806212 30379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:00:01.816254 30379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/krb5kdc/oryx_127.20.169.67.keytab
--principal=oryx/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:40777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=40319
--tserver_master_addrs=127.20.169.126:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:01.817592 30379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:01.818869 30379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:01.832537 30385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:01.832628 30384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:01.832597 30387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:01.833791 30379 server_base.cc:1034] running on GCE node
I20260927 05:00:01.834492 30379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:01.835543 30379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:01.836776 30379 hybrid_clock.cc:648] HybridClock initialized: now 1790485201836722 us; error 49 us; skew 500 ppm
I20260927 05:00:01.837136 30379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485201, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:01.840395 30379 init.cc:377] Logged in from keytab as oryx/127.20.169.67@KRBTEST.COM (short username oryx)
I20260927 05:00:01.842083 30379 webserver.cc:460] Webserver started at http://127.20.169.67:40319/ using document root <none> and password file <none>
I20260927 05:00:01.842834 30379 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:01.843006 30379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:01.849334 30379 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:00:01.852415 30394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:01.853951 30379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:00:01.854166 30379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "90465a43a08b4765b4cb812b26dd277f"
format_stamp: "Formatted at 2026-09-27 04:59:59 on dist-test-slave-39f2"
server_key: "41aa84bc819574441580ec523d0439a9"
server_key_iv: "8c6accc77e00b503ebe47c00571352a8"
server_key_version: "encryptionkey@0"
I20260927 05:00:01.855202 30379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:01.880640 30379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:01.885228 30379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:01.885535 30379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:01.886610 30379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:01.888019 30379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:01.888140 30379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:01.888273 30379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:01.888360 30379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:01.922287 30379 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:40777
I20260927 05:00:01.922376 30507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:40777 every 8 connection(s)
I20260927 05:00:01.923784 30379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:00:01.927219 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30379
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:00:01.943964 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.926632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:35835 (local address 127.20.169.126:42405)
0927 05:00:01.926879 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.926884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.928727 (+ 1843us) server_negotiation.cc:407] Connection header received
0927 05:00:01.932543 (+ 3816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.932554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.932678 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.932836 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.934782 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.935607 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.937960 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.938625 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.938773 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.943750 (+ 4977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:35835: BlockingRecv error: recv got EOF from 127.20.169.67:35835 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 05:00:01.944890 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.927315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:35835)
0927 05:00:01.928502 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.928578 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.932088 (+ 3510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.933218 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.933228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.933951 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.934613 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.934632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.935808 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.935814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.937816 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.937833 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.938866 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.938873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.939213 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.939887 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.939916 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.943618 (+ 3702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":482,"threads_started":1}
W20260927 05:00:01.945640 30508 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:42405 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.169.126:42405: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
I20260927 05:00:01.948767 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.928513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.20.169.126:42405)
0927 05:00:01.933076 (+ 4563us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.933085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.933643 (+ 558us) server_negotiation.cc:407] Connection header received
0927 05:00:01.933767 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.933772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.933834 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.933956 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.935505 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.936362 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.937736 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.938320 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.938471 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.941800 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:01.941830 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:01.941855 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:01.941923 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:01.945469 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:01.946276 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:01.946285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:01.946293 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:01.946495 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:01.946929 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:01.946932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:01.946934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:01.947358 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:01.947682 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:01.948029 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 05:00:01.948360 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4408,"thread_start_us":4188,"threads_started":1}
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:01.954833 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.946233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:40265)
0927 05:00:01.946407 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.946424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.946510 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.946797 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.946803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.947139 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.947271 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.947279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.948656 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.948661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.950069 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.950083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.951061 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.951065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.951183 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.951750 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.951768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.954564 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:00:01.954958 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.946322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:40265 (local address 127.20.169.126:42405)
0927 05:00:01.946546 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.946552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.946570 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:00:01.946616 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.946621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.946678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.946743 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.947410 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.948502 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.950237 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.950926 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.951046 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.954736 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:40265: BlockingRecv error: recv got EOF from 127.20.169.67:40265 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:01.962702 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.955731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:41053)
0927 05:00:01.955896 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.955914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.955986 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.956255 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.956258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.956536 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.956668 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.956676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.957694 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.957697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.958632 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.958642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.959250 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.959254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.959366 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.959921 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.959938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.962505 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:00:01.962929 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.955833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:41053 (local address 127.20.169.126:42405)
0927 05:00:01.956050 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.956054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.956067 (+ 13us) server_negotiation.cc:407] Connection header received
0927 05:00:01.956103 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.956106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.956151 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.956223 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.956785 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.957576 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.958763 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.959140 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.959228 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.962694 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:41053: BlockingRecv error: recv got EOF from 127.20.169.67:41053 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260927 05:00:01.963054 30508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:01 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:01.992547 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:01.983025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:39683)
0927 05:00:01.983575 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:01.983591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:01.983701 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:01.984039 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:01.984042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:01.984508 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:01.984698 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.984706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.986001 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.986005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.987060 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:01.987071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.987986 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.987992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:01.988170 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.988911 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:01.988935 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:01.992172 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":283,"threads_started":1}
W20260927 05:00:01.992658 30245 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:01.983125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:39683 (local address 127.20.169.126:42405)
0927 05:00:01.983285 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:01.983290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:01.983689 (+ 399us) server_negotiation.cc:407] Connection header received
0927 05:00:01.983818 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:01.983824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:01.983896 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:01.983989 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:01.984926 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.985859 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.987192 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:01.987763 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:01.987926 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:01.992415 (+ 4489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:39683: BlockingRecv error: recv got EOF from 127.20.169.65:39683 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 05:00:02 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:02 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:02.504717 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:02.495147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:60555)
0927 05:00:02.495688 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:02.495705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:02.495814 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:02.496275 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:02.496279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:02.496724 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:02.497019 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:02.497030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.498127 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.498130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:02.499083 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:02.499094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.499852 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.499860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:02.500068 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:02.500772 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:02.500844 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:02.504352 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":301,"threads_started":1}
W20260927 05:00:02.504926 30515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:02.495211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:60555 (local address 127.20.169.126:42405)
0927 05:00:02.495990 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:02.495995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:02.496010 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:00:02.496058 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:02.496062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:02.496117 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:02.496209 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:02.497163 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.497989 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.499210 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.499632 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.499725 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:02.504590 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:60555: BlockingRecv error: recv got EOF from 127.20.169.66:60555 (error 108)
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":590,"threads_started":1}
Sep 27 05:00:02 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:02 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:02.979036 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:02.963600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:47425)
0927 05:00:02.964202 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:02.964244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:02.964416 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:02.964839 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:02.964844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:02.965484 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:02.965694 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:02.965703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.966859 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.966865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:02.968073 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:02.968085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.968934 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.968941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:02.969142 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:02.975684 (+ 6542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:02.975703 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:02.978799 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":296,"threads_started":1}
W20260927 05:00:02.979437 30515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:02.963762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:47425 (local address 127.20.169.126:42405)
0927 05:00:02.964002 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:02.964009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:02.964388 (+ 379us) server_negotiation.cc:407] Connection header received
0927 05:00:02.964575 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:02.964581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:02.964657 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:02.964771 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:02.965857 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.966695 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.968228 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.968691 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.968890 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:02.979052 (+ 10162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:47425: BlockingRecv error: recv got EOF from 127.20.169.67:47425 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 05:00:02 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:02 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:03.002840 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:02.993600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:55677)
0927 05:00:02.994181 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:02.994198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:02.994366 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:02.994794 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:02.994799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:02.995240 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:02.995414 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:02.995424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.996568 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.996591 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:02.997681 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:02.997693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.998517 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.998524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:02.998703 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:02.999386 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:02.999411 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:03.002495 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":334,"threads_started":1}
W20260927 05:00:03.003019 30515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:02.993676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:55677 (local address 127.20.169.126:42405)
0927 05:00:02.993943 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:02.993957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:02.994341 (+ 384us) server_negotiation.cc:407] Connection header received
0927 05:00:02.994529 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:02.994533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:02.994612 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:02.994739 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:02.995564 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.996413 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.997862 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:02.998334 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:02.998496 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:03.002706 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:55677: BlockingRecv error: recv got EOF from 127.20.169.65:55677 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 05:00:03 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:03 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:03.514910 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:03.505723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:33725)
0927 05:00:03.506239 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:03.506257 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:03.506422 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:03.506889 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:03.506893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:03.507401 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:03.507580 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:03.507590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.508701 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.508705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:03.509842 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:03.509855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.510745 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.510752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:03.510937 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:03.511510 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:03.511529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:03.514542 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":283,"threads_started":1}
W20260927 05:00:03.515053 30519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:03.505854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:33725 (local address 127.20.169.126:42405)
0927 05:00:03.506597 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:03.506601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:03.506616 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:00:03.506663 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:03.506667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:03.506736 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:03.506828 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:03.507739 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.508534 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.510002 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.510564 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.510716 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:03.514765 (+ 4049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:33725: BlockingRecv error: recv got EOF from 127.20.169.66:33725 (error 108)
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":538,"threads_started":1}
Sep 27 05:00:03 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:03 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:03.988906 30520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:03.980028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:60005)
0927 05:00:03.980586 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:03.980605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:03.980767 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:03.981188 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:03.981192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:03.981774 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:03.982012 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:03.982025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.983281 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.983285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:03.984214 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:03.984225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.984936 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.984944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:03.985060 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:03.985793 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:03.985812 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:03.988640 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
W20260927 05:00:03.989176 30519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:03.980277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:60005 (local address 127.20.169.126:42405)
0927 05:00:03.980510 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:03.980516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:03.980691 (+ 175us) server_negotiation.cc:407] Connection header received
0927 05:00:03.980946 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:03.980950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:03.981013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:03.981096 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:03.982177 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.983149 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.984352 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:03.984735 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:03.984887 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:03.988899 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:60005: BlockingRecv error: recv got EOF from 127.20.169.67:60005 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 05:00:04 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:04 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:04.012677 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:04.003812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:59621)
0927 05:00:04.004401 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:04.004427 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:04.004538 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:04.004980 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:04.004987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:04.005438 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:04.005622 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:04.005632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.006815 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.006827 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:04.007796 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:04.007809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.008573 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.008580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:04.008779 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:04.009505 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:04.009526 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:04.012312 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":332,"threads_started":1}
W20260927 05:00:04.012768 30519 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:04.003926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:59621 (local address 127.20.169.126:42405)
0927 05:00:04.004090 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:04.004094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:04.004499 (+ 405us) server_negotiation.cc:407] Connection header received
0927 05:00:04.004668 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:04.004673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:04.004750 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:04.004919 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:04.005792 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.006649 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.007950 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.008411 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.008562 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:04.012505 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:59621: BlockingRecv error: recv got EOF from 127.20.169.65:59621 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 05:00:04 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:04 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:04.525305 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:04.516142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:42773)
0927 05:00:04.517294 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:04.517328 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:04.517460 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:04.517894 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:04.517897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:04.518236 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:04.518409 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:04.518417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.519465 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.519468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:04.520492 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:04.520504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.521193 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.521198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:04.521404 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:04.522094 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:04.522112 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:04.525061 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":877,"threads_started":1}
W20260927 05:00:04.525521 30523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:04.516321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:42773 (local address 127.20.169.126:42405)
0927 05:00:04.517608 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:04.517612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:04.517628 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:00:04.517673 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:04.517677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:04.517741 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:04.517839 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:04.518550 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.519337 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.520617 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.521070 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.521195 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:04.525237 (+ 4042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:42773: BlockingRecv error: recv got EOF from 127.20.169.66:42773 (error 108)
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":1085,"threads_started":1}
Sep 27 05:00:04 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:04 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:04.998518 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:04.989933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:36143)
0927 05:00:04.990584 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:04.990601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:04.990763 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:04.991146 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:04.991152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:04.991646 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:04.991811 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:04.991828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.992992 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.993001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:04.993896 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:04.993907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.994612 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.994618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:04.994788 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:04.995417 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:04.995435 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:04.998151 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":408,"threads_started":1}
W20260927 05:00:04.998588 30523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:04.990041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:36143 (local address 127.20.169.126:42405)
0927 05:00:04.990408 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:04.990414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:04.990690 (+ 276us) server_negotiation.cc:407] Connection header received
0927 05:00:04.990923 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:04.990930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:04.991004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:04.991111 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:04.992012 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.992789 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.994046 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:04.994460 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:04.994590 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:04.998361 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:36143: BlockingRecv error: recv got EOF from 127.20.169.67:36143 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Sep 27 05:00:05 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:05 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:05.023218 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:05.013704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:39589)
0927 05:00:05.014244 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:05.014260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:05.014381 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:05.014765 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:05.014769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:05.015270 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:05.015529 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:05.015546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.016915 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.016919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:05.017906 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:05.017918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.018704 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.018710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:05.018858 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:05.019498 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:05.019518 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:05.022972 (+ 3454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":292,"threads_started":1}
W20260927 05:00:05.023440 30523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:05.013791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:39589 (local address 127.20.169.126:42405)
0927 05:00:05.014006 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:05.014011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:05.014390 (+ 379us) server_negotiation.cc:407] Connection header received
0927 05:00:05.014501 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:05.014505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:05.014592 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:05.014689 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:05.015701 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.016735 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.018057 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.018517 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.018642 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:05.023179 (+ 4537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:39589: BlockingRecv error: recv got EOF from 127.20.169.65:39589 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 05:00:05 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:05 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:05.534600 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:05.526331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:34117)
0927 05:00:05.526831 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:05.526848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:05.526959 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:05.527305 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:05.527309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:05.527688 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:05.527890 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:05.527902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.528962 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.528966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:05.529977 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:05.530027 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.530808 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.530812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:05.530943 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:05.531561 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:05.531578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:05.534375 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
W20260927 05:00:05.535322 30527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:05.526383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:34117 (local address 127.20.169.126:42405)
0927 05:00:05.526827 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:05.526832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:05.526941 (+ 109us) server_negotiation.cc:407] Connection header received
0927 05:00:05.527058 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:05.527063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:05.527141 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:05.527236 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:05.528028 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.528766 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.530168 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:05.530684 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:05.530777 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:05.534631 (+ 3854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:34117: BlockingRecv error: recv got EOF from 127.20.169.66:34117 (error 108)
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":253,"threads_started":1}
Sep 27 05:00:06 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:06 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:06.007808 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:05.999358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:56285)
0927 05:00:05.999837 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:05.999855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:06.000016 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:06.000326 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:06.000329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:06.000697 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:06.000917 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:06.000926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.001985 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.001990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:06.003128 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:06.003141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.003864 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.003869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:06.003999 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:06.004637 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:06.004656 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:06.007578 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"spinlock_wait_cycles":32128,"thread_start_us":350,"threads_started":1}
W20260927 05:00:06.008076 30527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:05.999459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56285 (local address 127.20.169.126:42405)
0927 05:00:05.999700 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:05.999706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:05.999951 (+ 245us) server_negotiation.cc:407] Connection header received
0927 05:00:06.000123 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:06.000127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:06.000190 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:06.000271 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:06.001041 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.001801 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.003298 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.003705 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.003800 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:06.007780 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:56285: BlockingRecv error: recv got EOF from 127.20.169.67:56285 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 05:00:06 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:06 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:06.033084 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:06.024256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:42617)
0927 05:00:06.024862 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:06.024890 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:06.025038 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:06.025461 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:06.025466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:06.025877 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:06.026047 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:06.026056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.027106 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.027112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:06.028031 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:06.028042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.028834 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.028841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:06.029019 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:06.029687 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:06.029705 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:06.032666 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":238,"threads_started":1}
W20260927 05:00:06.033154 30527 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:06.024348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:42617 (local address 127.20.169.126:42405)
0927 05:00:06.024540 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:06.024549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:06.024972 (+ 423us) server_negotiation.cc:407] Connection header received
0927 05:00:06.025174 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:06.025179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:06.025269 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:06.025374 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:06.026209 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.026963 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.028175 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.028640 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.028785 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:06.032913 (+ 4128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:42617: BlockingRecv error: recv got EOF from 127.20.169.65:42617 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 05:00:06 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:06 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:06.544801 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:06.535579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:37349)
0927 05:00:06.536150 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:06.536168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:06.536271 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:06.536682 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:06.536686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:06.537179 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:06.537397 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:06.537408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.538491 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.538496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:06.539643 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:06.539658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.540414 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.540419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:06.540552 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:06.541255 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:06.541321 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:06.544458 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
W20260927 05:00:06.545128 30531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:06.535748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:37349 (local address 127.20.169.126:42405)
0927 05:00:06.536194 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:06.536198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:06.536214 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:00:06.536443 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:06.536447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:06.536507 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:06.536599 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:06.537585 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.538302 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.539796 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:06.540232 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:06.540361 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:06.544750 (+ 4389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:37349: BlockingRecv error: recv got EOF from 127.20.169.66:37349 (error 108)
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":258,"threads_started":1}
Sep 27 05:00:07 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:07 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:07.021592 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:07.008866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:48015)
0927 05:00:07.009419 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:07.009437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:07.009552 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:07.009851 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:07.009854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:07.010262 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:07.010444 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:07.010453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.011468 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.011472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:07.012399 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:07.012411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.013165 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.013169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:07.013284 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:07.017618 (+ 4334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:07.017651 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:07.021125 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":289,"threads_started":1}
W20260927 05:00:07.021695 30531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:07.009010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:48015 (local address 127.20.169.126:42405)
0927 05:00:07.009249 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:07.009254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:07.009537 (+ 283us) server_negotiation.cc:407] Connection header received
0927 05:00:07.009660 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:07.009664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:07.009726 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:07.009814 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:07.010619 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.011354 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.012562 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.013033 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.013145 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:07.021373 (+ 8228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:48015: BlockingRecv error: recv got EOF from 127.20.169.67:48015 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 05:00:07 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:07 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:07.042403 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:07.034126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:47599)
0927 05:00:07.034679 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:07.034696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:07.034810 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:07.035169 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:07.035173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:07.035600 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:07.035753 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:07.035763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.036862 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.036873 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:07.037802 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:07.037813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.038483 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.038487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:07.038608 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:07.039263 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:07.039282 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:07.042140 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":310,"threads_started":1}
W20260927 05:00:07.042752 30531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:07.034176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:47599 (local address 127.20.169.126:42405)
0927 05:00:07.034382 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:07.034387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:07.034770 (+ 383us) server_negotiation.cc:407] Connection header received
0927 05:00:07.034917 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:07.034921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:07.035004 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:07.035089 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:07.035918 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.036646 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.037930 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.038329 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.038427 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:07.042355 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:47599: BlockingRecv error: recv got EOF from 127.20.169.65:47599 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 05:00:07 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:07 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:07.554931 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:07.545854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:58803)
0927 05:00:07.546335 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:07.546353 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:07.546460 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:07.547055 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:07.547059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:07.547468 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:07.547628 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:07.547638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.548725 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.548729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:07.549713 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:07.549726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.550548 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.550554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:07.550728 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:07.551497 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:07.551522 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:07.554682 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":270,"threads_started":1}
W20260927 05:00:07.555388 30535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:07.545915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:58803 (local address 127.20.169.126:42405)
0927 05:00:07.546746 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:07.546750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:07.546765 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:00:07.546812 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:07.546816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:07.546884 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:07.546972 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:07.547783 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.548574 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.549876 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:07.550362 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:07.550461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:07.554931 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:58803: BlockingRecv error: recv got EOF from 127.20.169.66:58803 (error 108)
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":645,"threads_started":1}
Sep 27 05:00:08 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:08 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:08.038187 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:08.022691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:52673)
0927 05:00:08.023219 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:08.023235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:08.023451 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:08.023790 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:08.023795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:08.024223 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:08.024407 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:08.024416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.025554 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.025559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:08.026577 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:08.026589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.027290 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.027295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:08.027433 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:08.034504 (+ 7071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:08.034524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:08.037814 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
W20260927 05:00:08.038293 30535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:08.022782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:52673 (local address 127.20.169.126:42405)
0927 05:00:08.023027 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:08.023035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:08.023375 (+ 340us) server_negotiation.cc:407] Connection header received
0927 05:00:08.023589 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:08.023598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:08.023660 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:08.023764 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:08.024564 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.025387 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.026722 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.027145 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.027246 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:08.038036 (+ 10790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:52673: BlockingRecv error: recv got EOF from 127.20.169.67:52673 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 05:00:08 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:08 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:08.052341 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:08.043431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:33319)
0927 05:00:08.043983 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:08.044000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:08.044112 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:08.044498 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:08.044501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:08.044963 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:08.045140 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:08.045150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.046207 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.046210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:08.047181 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:08.047193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.047915 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.047919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:08.048034 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:08.048620 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:08.048639 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:08.052078 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":241,"threads_started":1}
W20260927 05:00:08.052640 30535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:08.043489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:33319 (local address 127.20.169.126:42405)
0927 05:00:08.043731 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:08.043742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:08.044083 (+ 341us) server_negotiation.cc:407] Connection header received
0927 05:00:08.044248 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:08.044252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:08.044347 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:08.044441 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:08.045330 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.046073 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.047324 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.047738 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.047837 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:08.052347 (+ 4510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:33319: BlockingRecv error: recv got EOF from 127.20.169.65:33319 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 05:00:08 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:08 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:08.564900 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:08.555833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:45059)
0927 05:00:08.556480 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:08.556498 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:08.556623 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:08.557046 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:08.557050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:08.557494 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:08.557660 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:08.557668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.558751 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.558755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:08.559869 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:08.559883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.560577 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.560582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:08.560700 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:08.561345 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:08.561363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:08.564575 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":264,"threads_started":1}
W20260927 05:00:08.565120 30539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:08.555998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:45059 (local address 127.20.169.126:42405)
0927 05:00:08.556475 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:08.556479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:08.556579 (+ 100us) server_negotiation.cc:407] Connection header received
0927 05:00:08.556731 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:08.556735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:08.556882 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:08.556975 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:08.557786 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.558614 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.560009 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:08.560434 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:08.560529 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:08.564758 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:45059: BlockingRecv error: recv got EOF from 127.20.169.66:45059 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":265,"threads_started":1}
Sep 27 05:00:09 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:09 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:09.047863 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:09.039147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:56083)
0927 05:00:09.039800 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:09.039816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:09.039948 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:09.040259 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:09.040262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:09.040648 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:09.040860 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:09.040870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.041914 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.041918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:09.042817 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:09.042828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.043479 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.043483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:09.043590 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:09.044252 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:09.044276 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:09.047419 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":336,"threads_started":1}
W20260927 05:00:09.047897 30539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:09.039221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56083 (local address 127.20.169.126:42405)
0927 05:00:09.039409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:09.039414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:09.039894 (+ 480us) server_negotiation.cc:407] Connection header received
0927 05:00:09.040052 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:09.040055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:09.040122 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:09.040214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:09.040999 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.041777 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.042956 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.043353 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.043444 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:09.047639 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:56083: BlockingRecv error: recv got EOF from 127.20.169.67:56083 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 05:00:09 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:09 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:09.061781 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:09.053431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:58545)
0927 05:00:09.053941 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:09.053957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:09.054065 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:09.054365 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:09.054369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:09.054732 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:09.054875 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:09.054883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.055993 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.055998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:09.057194 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:09.057215 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.058062 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.058066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:09.058178 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:09.058784 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:09.058801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:09.061557 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
W20260927 05:00:09.062142 30539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:09.053650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:58545 (local address 127.20.169.126:42405)
0927 05:00:09.053840 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:09.053846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:09.054031 (+ 185us) server_negotiation.cc:407] Connection header received
0927 05:00:09.054162 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:09.054166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:09.054238 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:09.054366 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:09.054991 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.055829 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.057325 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.057904 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.057999 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:09.061769 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:58545: BlockingRecv error: recv got EOF from 127.20.169.65:58545 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 05:00:09 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:09 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:09.574517 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:09.565815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:59211)
0927 05:00:09.566374 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:09.566391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:09.566495 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:09.566972 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:09.566976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:09.567400 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:09.567549 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:09.567558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.568733 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.568737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:09.569720 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:09.569731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.570490 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.570494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:09.570604 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:09.571305 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:09.571327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:09.574272 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
W20260927 05:00:09.574709 30543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:09.565986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:59211 (local address 127.20.169.126:42405)
0927 05:00:09.566664 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:09.566668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:09.566683 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:00:09.566728 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:09.566732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:09.566799 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:09.566916 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:09.567690 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.568597 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.569886 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:09.570326 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:09.570427 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:09.574475 (+ 4048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:59211: BlockingRecv error: recv got EOF from 127.20.169.66:59211 (error 108)
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":480,"threads_started":1}
Sep 27 05:00:10 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:10 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:10.057446 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:10.048966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:60283)
0927 05:00:10.049491 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:10.049507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:10.049663 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:10.050005 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:10.050010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:10.050456 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:10.050606 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:10.050615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.051688 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.051700 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:10.052598 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:10.052609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.053434 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.053440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:10.053610 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:10.054273 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:10.054291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:10.057218 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
W20260927 05:00:10.057664 30543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:10.049007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:60283 (local address 127.20.169.126:42405)
0927 05:00:10.049189 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:10.049197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:10.049594 (+ 397us) server_negotiation.cc:407] Connection header received
0927 05:00:10.049785 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:10.049791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:10.049864 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:10.049978 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:10.050761 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.051538 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.052743 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.053223 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.053386 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:10.057437 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:60283: BlockingRecv error: recv got EOF from 127.20.169.67:60283 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 05:00:10 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:10 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:10.071203 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:10.062612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:50421)
0927 05:00:10.063107 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:10.063123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:10.063273 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:10.063642 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:10.063647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:10.064034 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:10.064194 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:10.064203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.065295 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.065300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:10.066240 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:10.066251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.066986 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.066992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:10.067168 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:10.067713 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:10.067730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:10.070888 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
W20260927 05:00:10.071337 30543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:10.062677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:50421 (local address 127.20.169.126:42405)
0927 05:00:10.062876 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:10.062885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:10.063205 (+ 320us) server_negotiation.cc:407] Connection header received
0927 05:00:10.063420 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:10.063435 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:10.063504 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:10.063605 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:10.064335 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.065157 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.066407 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.066835 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.066971 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:10.071074 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:50421: BlockingRecv error: recv got EOF from 127.20.169.65:50421 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 05:00:10 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:10 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:10.584266 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:10.575463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:40019)
0927 05:00:10.576040 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:10.576057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:10.576173 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:10.576529 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:10.576533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:10.576976 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:10.577141 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:10.577151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.578215 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.578220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:10.579155 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:10.579166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.579916 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.579921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:10.580042 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:10.580646 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:10.580665 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:10.583971 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":284,"threads_started":1}
W20260927 05:00:10.584590 30547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:10.575555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40019 (local address 127.20.169.126:42405)
0927 05:00:10.576067 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:10.576071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:10.576111 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:00:10.576283 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:10.576287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:10.576369 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:10.576474 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:10.577278 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.578055 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.579295 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:10.579751 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:10.579865 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:10.584252 (+ 4387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:40019: BlockingRecv error: recv got EOF from 127.20.169.66:40019 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":273,"threads_started":1}
Sep 27 05:00:11 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:11 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:11.067981 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:11.058431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:49085)
0927 05:00:11.059018 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:11.059035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:11.059208 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:11.059618 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:11.059624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:11.060074 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:11.060265 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:11.060278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.061436 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.061442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:11.062453 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:11.062465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.063216 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.063222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:11.063426 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:11.064062 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:11.064083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:11.067629 (+ 3546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":328,"threads_started":1}
W20260927 05:00:11.068164 30547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:11.058504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:49085 (local address 127.20.169.126:42405)
0927 05:00:11.058772 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:11.058779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:11.059127 (+ 348us) server_negotiation.cc:407] Connection header received
0927 05:00:11.059373 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:11.059380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:11.059458 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:11.059581 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:11.060437 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.061287 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.062612 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.063045 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.063198 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:11.067919 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:49085: BlockingRecv error: recv got EOF from 127.20.169.67:49085 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 27 05:00:11 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:11 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:11.082201 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:11.072113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:33419)
0927 05:00:11.072742 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:11.072762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:11.072973 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:11.073480 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:11.073484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:11.073855 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:11.074016 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:11.074026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.075101 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.075105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:11.076076 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:11.076088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.076879 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.076886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:11.077062 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:11.077803 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:11.077832 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:11.081778 (+ 3946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":424,"threads_started":1}
W20260927 05:00:11.083063 30547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:11.072235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:33419 (local address 127.20.169.126:42405)
0927 05:00:11.072401 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:11.072406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:11.072969 (+ 563us) server_negotiation.cc:407] Connection header received
0927 05:00:11.073112 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:11.073116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:11.073239 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:11.073400 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:11.074167 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.074967 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.076225 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.076658 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.076759 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:11.082689 (+ 5930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:33419: BlockingRecv error: recv got EOF from 127.20.169.65:33419 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 05:00:11 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:11 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:11.594766 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:11.585480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:39675)
0927 05:00:11.586061 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:11.586078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:11.586186 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:11.586595 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:11.586599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:11.586977 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:11.587195 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:11.587205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.588386 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.588390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:11.589544 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:11.589559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.590826 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.590833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:11.591027 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:11.591698 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:11.591716 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:11.594535 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":308,"threads_started":1}
W20260927 05:00:11.595073 30551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:11.585603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:39675 (local address 127.20.169.126:42405)
0927 05:00:11.586159 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:11.586164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:11.586183 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:00:11.586322 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:11.586326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:11.586406 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:11.586507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:11.587403 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.588218 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.589702 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:11.590616 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:11.590754 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:11.594755 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:39675: BlockingRecv error: recv got EOF from 127.20.169.66:39675 (error 108)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":313,"threads_started":1}
Sep 27 05:00:12 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:12 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:12.078238 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:12.069013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:45685)
0927 05:00:12.069524 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:12.069538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:12.069690 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:12.070164 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:12.070170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:12.070788 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:12.071057 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:12.071074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.072462 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.072467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:12.073558 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:12.073570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.074240 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.074244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:12.074406 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:12.075020 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:12.075039 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:12.077999 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
W20260927 05:00:12.078431 30551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:12.069137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:45685 (local address 127.20.169.126:42405)
0927 05:00:12.069372 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:12.069378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:12.069620 (+ 242us) server_negotiation.cc:407] Connection header received
0927 05:00:12.069811 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:12.069817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:12.069959 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:12.070085 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:12.071257 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.072279 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.073708 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.074100 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.074196 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:12.078214 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:45685: BlockingRecv error: recv got EOF from 127.20.169.67:45685 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 05:00:12 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:12 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:12.091985 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:12.083283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:54093)
0927 05:00:12.083820 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:12.083837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:12.083974 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:12.084286 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:12.084292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:12.084705 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:12.084985 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:12.084994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.086098 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.086101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:12.087307 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:12.087319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.087972 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.087977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:12.088107 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:12.088674 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:12.088690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:12.091618 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
W20260927 05:00:12.092083 30551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:12.083347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:54093 (local address 127.20.169.126:42405)
0927 05:00:12.083533 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:12.083539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:12.083925 (+ 386us) server_negotiation.cc:407] Connection header received
0927 05:00:12.084071 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:12.084075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:12.084142 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:12.084222 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:12.085169 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.085985 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.087435 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.087799 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.087922 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:12.091806 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:54093: BlockingRecv error: recv got EOF from 127.20.169.65:54093 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 05:00:12 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:12 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:12.604612 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:12.595682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:37301)
0927 05:00:12.596239 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:12.596256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:12.596377 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:12.596696 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:12.596699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:12.597149 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:12.597359 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:12.597370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.599057 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.599063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:12.600057 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:12.600088 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.600886 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.600899 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:12.601027 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:12.601598 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:12.601615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:12.604370 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
W20260927 05:00:12.605124 30555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:12.595754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:37301 (local address 127.20.169.126:42405)
0927 05:00:12.596238 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:12.596242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:12.596337 (+ 95us) server_negotiation.cc:407] Connection header received
0927 05:00:12.596476 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:12.596481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:12.596555 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:12.596660 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:12.597582 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.598892 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.600229 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:12.600674 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:12.600773 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:12.604784 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:37301: BlockingRecv error: recv got EOF from 127.20.169.66:37301 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":263,"threads_started":1}
Sep 27 05:00:13 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:13 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:13.090739 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:13.079185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:56155)
0927 05:00:13.079718 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:13.079734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:13.079877 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:13.080225 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:13.080228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:13.080657 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:13.080874 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:13.080884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.082014 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.082018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:13.082910 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:13.082921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.083691 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.083695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:13.083813 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:13.087304 (+ 3491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:13.087321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:13.090357 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":301,"threads_started":1}
W20260927 05:00:13.090883 30555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:13.079342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56155 (local address 127.20.169.126:42405)
0927 05:00:13.079536 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:13.079541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:13.079828 (+ 287us) server_negotiation.cc:407] Connection header received
0927 05:00:13.080024 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:13.080029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:13.080094 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:13.080179 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:13.081037 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.081881 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.083082 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.083546 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.083649 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:13.090597 (+ 6948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:56155: BlockingRecv error: recv got EOF from 127.20.169.67:56155 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 05:00:13 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:13 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:13.102103 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:13.092953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:52175)
0927 05:00:13.093592 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:13.093612 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:13.093723 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:13.094079 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:13.094082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:13.094441 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:13.094587 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:13.094595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.095619 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.095624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:13.096544 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:13.096555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.097396 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.097401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:13.097554 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:13.098135 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:13.098154 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:13.101704 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":426,"threads_started":1}
W20260927 05:00:13.102233 30555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:13.092979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:52175 (local address 127.20.169.126:42405)
0927 05:00:13.093199 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:13.093208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:13.093691 (+ 483us) server_negotiation.cc:407] Connection header received
0927 05:00:13.093867 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:13.093872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:13.093937 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:13.094031 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:13.094735 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.095483 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.096704 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.097229 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.097375 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:13.101953 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:52175: BlockingRecv error: recv got EOF from 127.20.169.65:52175 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 05:00:13 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:13 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:13.614604 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:13.605541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:50463)
0927 05:00:13.606138 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:13.606155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:13.606277 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:13.606839 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:13.606844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:13.607238 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:13.607413 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:13.607421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.608477 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.608482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:13.609673 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:13.609688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.610460 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.610465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:13.610609 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:13.611185 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:13.611203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:13.614261 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":377,"threads_started":1}
W20260927 05:00:13.614729 30559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:13.605642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:50463 (local address 127.20.169.126:42405)
0927 05:00:13.606103 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:13.606108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:13.606235 (+ 127us) server_negotiation.cc:407] Connection header received
0927 05:00:13.606540 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:13.606546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:13.606651 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:13.606754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:13.607542 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.608298 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.609809 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:13.610288 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:13.610392 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:13.614477 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:50463: BlockingRecv error: recv got EOF from 127.20.169.66:50463 (error 108)
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":241,"threads_started":1}
Sep 27 05:00:14 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:14 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:14.100567 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:14.091713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:56609)
0927 05:00:14.092237 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:14.092253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:14.092428 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:14.092756 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:14.092761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:14.093200 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:14.093360 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:14.093368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.094408 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.094414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:14.095321 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:14.095331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.096010 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.096016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:14.096199 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:14.096915 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:14.096946 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:14.100315 (+ 3369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":315,"threads_started":1}
W20260927 05:00:14.100776 30559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:14.091757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56609 (local address 127.20.169.126:42405)
0927 05:00:14.091960 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:14.091969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:14.092342 (+ 373us) server_negotiation.cc:407] Connection header received
0927 05:00:14.092562 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:14.092567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:14.092626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:14.092724 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:14.093527 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.094260 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.095481 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.095860 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.095989 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:14.100551 (+ 4562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:56609: BlockingRecv error: recv got EOF from 127.20.169.67:56609 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 05:00:14 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:14 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:14.111953 30561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:14.103372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:40925)
0927 05:00:14.103869 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:14.103884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:14.104032 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:14.104413 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:14.104418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:14.104846 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:14.105018 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:14.105035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.106024 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.106032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:14.106984 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:14.106995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.107707 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.107713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:14.107886 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:14.108515 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:14.108540 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:14.111563 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
W20260927 05:00:14.112025 30559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:14.103378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:40925 (local address 127.20.169.126:42405)
0927 05:00:14.103588 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:14.103595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:14.103968 (+ 373us) server_negotiation.cc:407] Connection header received
0927 05:00:14.104147 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:14.104152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:14.104230 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:14.104386 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:14.105159 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.105880 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.107126 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.107522 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.107667 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:14.111753 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:40925: BlockingRecv error: recv got EOF from 127.20.169.65:40925 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:00:14 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:14 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:14.624758 30562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:14.615682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:53217)
0927 05:00:14.616303 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:14.616323 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:14.616439 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:14.616992 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:14.616997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:14.617441 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:14.617604 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:14.617614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.618663 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.618667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:14.619710 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:14.619724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.620499 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.620507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:14.620653 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:14.621347 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:14.621367 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:14.624381 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":293,"threads_started":1}
W20260927 05:00:14.625011 30563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:14.615859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:53217 (local address 127.20.169.126:42405)
0927 05:00:14.616535 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:14.616540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:14.616555 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:00:14.616605 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:14.616609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:14.616677 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:14.616775 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:14.617746 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.618524 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.619865 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:14.620274 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:14.620429 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:14.624676 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:53217: BlockingRecv error: recv got EOF from 127.20.169.66:53217 (error 108)
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":496,"threads_started":1}
Sep 27 05:00:15 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:15 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:15.110625 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:15.101561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:38911)
0927 05:00:15.102122 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:15.102139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:15.102256 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:15.102572 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:15.102575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:15.102997 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:15.103153 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:15.103162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.104215 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.104219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:15.105271 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:15.105283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.106082 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.106091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:15.106328 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:15.107191 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:15.107211 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:15.110369 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":305,"threads_started":1}
W20260927 05:00:15.110716 30563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:15.101639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:38911 (local address 127.20.169.126:42405)
0927 05:00:15.101803 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:15.101807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:15.102246 (+ 439us) server_negotiation.cc:407] Connection header received
0927 05:00:15.102368 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:15.102372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:15.102437 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:15.102527 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:15.103302 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.104073 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.105412 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.105822 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.105921 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:15.110536 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:38911: BlockingRecv error: recv got EOF from 127.20.169.67:38911 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 05:00:15 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:15 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:15.121502 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:15.112961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:35267)
0927 05:00:15.113519 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:15.113534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:15.113685 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:15.114044 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:15.114049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:15.114419 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:15.114586 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:15.114595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.115619 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.115625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:15.116549 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:15.116561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.117319 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.117325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:15.117504 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:15.118112 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:15.118136 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:15.121245 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":309,"threads_started":1}
W20260927 05:00:15.121894 30563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:15.113026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:35267 (local address 127.20.169.126:42405)
0927 05:00:15.113291 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:15.113300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:15.113626 (+ 326us) server_negotiation.cc:407] Connection header received
0927 05:00:15.113807 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:15.113812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:15.113898 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:15.114004 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:15.114736 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.115476 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.116696 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.117168 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.117297 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:15.121497 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:35267: BlockingRecv error: recv got EOF from 127.20.169.65:35267 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 27 05:00:15 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:15 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:15.633863 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:15.625688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:37329)
0927 05:00:15.626167 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:15.626195 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:15.626299 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:15.626620 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:15.626623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:15.626972 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:15.627136 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:15.627145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.628116 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.628120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:15.629116 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:15.629128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.629861 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.629866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:15.629998 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:15.630593 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:15.630614 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:15.633580 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
W20260927 05:00:15.634021 30567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:15.625755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:37329 (local address 127.20.169.126:42405)
0927 05:00:15.626215 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:15.626219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:15.626238 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:00:15.626423 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:15.626427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:15.626496 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:15.626581 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:15.627261 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.627978 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.629233 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:15.629739 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:15.629837 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:15.633789 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:37329: BlockingRecv error: recv got EOF from 127.20.169.66:37329 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
Sep 27 05:00:16 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:16 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.67@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:16.120026 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:16.111694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.67:37167)
0927 05:00:16.112244 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:16.112260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:16.112368 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:16.112659 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:16.112662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:16.113104 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:16.113278 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:16.113288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.114288 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.114291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:16.115170 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:16.115180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.115824 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.115827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:16.115943 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.116591 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:16.116617 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:16.119654 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":312,"threads_started":1}
W20260927 05:00:16.120064 30567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:16.111808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:37167 (local address 127.20.169.126:42405)
0927 05:00:16.111977 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:16.111981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:16.112339 (+ 358us) server_negotiation.cc:407] Connection header received
0927 05:00:16.112465 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:16.112469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:16.112533 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:16.112618 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:16.113427 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.114165 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.115299 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.115707 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.115810 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.119845 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.67:37167: BlockingRecv error: recv got EOF from 127.20.169.67:37167 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 05:00:16 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:16 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.65@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:16.132232 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:16.122507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.65:39117)
0927 05:00:16.123004 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:16.123018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:16.123165 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:16.123515 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:16.123520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:16.123912 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:16.124075 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:16.124083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.125595 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.125602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:16.126740 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:16.126753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.127529 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.127535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:16.127719 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.128465 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:16.128494 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:16.131990 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":243,"threads_started":1}
W20260927 05:00:16.132637 30567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:16.122559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:39117 (local address 127.20.169.126:42405)
0927 05:00:16.122752 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:16.122759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:16.123095 (+ 336us) server_negotiation.cc:407] Connection header received
0927 05:00:16.123289 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:16.123293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:16.123361 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:16.123489 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:16.124209 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.125438 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.126907 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.127361 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.127501 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.132247 (+ 4746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.65:39117: BlockingRecv error: recv got EOF from 127.20.169.65:39117 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 05:00:16 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:00:16 dist-test-slave-39f2 krb5kdc[29503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.169.66@KRBTEST.COM for oryx/127.20.169.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:00:16.644291 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:16.635204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:42405 (local address 127.20.169.66:45493)
0927 05:00:16.635871 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:16.635892 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:16.636069 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:16.636497 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:16.636500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:16.637080 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:16.637273 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:16.637284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.638394 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.638398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:16.639353 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:16.639365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.640095 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.640100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:16.640235 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.640960 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:16.640980 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:16.644053 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.169.126:42405: Server oryx/127.20.169.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":376,"threads_started":1}
W20260927 05:00:16.644604 30571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:16.635275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:45493 (local address 127.20.169.126:42405)
0927 05:00:16.635808 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:16.635822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:16.635980 (+ 158us) server_negotiation.cc:407] Connection header received
0927 05:00:16.636206 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:16.636217 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:16.636298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:16.636430 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:16.637437 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.638201 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.639515 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.639970 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.640062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.644278 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.169.66:45493: BlockingRecv error: recv got EOF from 127.20.169.66:45493 (error 108)
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
I20260927 05:00:16.938043 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30095
I20260927 05:00:16.950413 30573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:16.939123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.20.169.65:35747)
0927 05:00:16.939789 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:16.939796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:16.939828 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:00:16.939885 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:16.939891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:16.940134 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:16.940352 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:16.941884 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.942923 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.944142 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.944698 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.944870 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.947315 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:16.947326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:16.947344 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:16.947376 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:16.949729 (+ 2353us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.169.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:00:16.950083 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46588: Request ticket server kudu/127.20.169.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":338,"threads_started":1}
W20260927 05:00:16.950531 30573 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46588: Request ticket server kudu/127.20.169.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:00:16.950495 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30095: Not authorized: Client connection negotiation failed: client connection to 127.20.169.65:35747: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.169.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:00:16.950708 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30095
I20260927 05:00:16.969043 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30247
I20260927 05:00:16.981696 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:16.970131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.20.169.66:32959)
0927 05:00:16.970748 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:16.970755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:16.970787 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:00:16.970848 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:16.970854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:16.971074 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:16.971294 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:16.972650 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.973807 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.975070 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:16.975669 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:16.975810 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:16.978430 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:16.978440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:16.978454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:16.978491 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:16.981022 (+ 2531us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.169.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:00:16.981333 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40262: Request ticket server kudu/127.20.169.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":313,"threads_started":1}
W20260927 05:00:16.981693 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30247: Not authorized: Client connection negotiation failed: client connection to 127.20.169.66:32959: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.169.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:00:16.981808 30574 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40262: Request ticket server kudu/127.20.169.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:00:16.981887 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30247
I20260927 05:00:16.999222 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30379
I20260927 05:00:17.012339 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:17.000206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.20.169.67:40777)
0927 05:00:17.000896 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:17.000903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:17.000932 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:00:17.000994 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:17.001000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:17.001252 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:17.001459 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:17.002917 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:17.004054 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:17.005505 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:17.006116 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:17.006270 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:17.009009 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:17.009018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:17.009032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:17.009072 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:17.011609 (+ 2537us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.169.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:00:17.011962 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59212: Request ticket server kudu/127.20.169.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":317,"threads_started":1}
W20260927 05:00:17.012360 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30379: Not authorized: Client connection negotiation failed: client connection to 127.20.169.67:40777: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.169.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:00:17.012458 30575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59212: Request ticket server kudu/127.20.169.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:00:17.012504 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30379
I20260927 05:00:17.032045 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30025
I20260927 05:00:17.033473 30088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:17.150547 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30025
2026-09-27T05:00:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21247 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T05:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:00:17Z Disabled control of system clock
I20260927 05:00:17.231292 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35885
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 05:00:17.569274 30580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:17.569566 30580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:17.569698 30580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:17.577045 30580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:00:17.577148 30580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:17.577217 30580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:17.577311 30580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:00:17.577369 30580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:00:17.586704 30580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:17.588001 30580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:17.589790 30580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:17.602147 30586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:17.602234 30587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:17.602198 30589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:17.602962 30580 server_base.cc:1034] running on GCE node
I20260927 05:00:17.603715 30580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:17.605055 30580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:17.606288 30580 hybrid_clock.cc:648] HybridClock initialized: now 1790485217606237 us; error 43 us; skew 500 ppm
I20260927 05:00:17.606696 30580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:17.608928 30580 webserver.cc:460] Webserver started at http://127.20.169.126:37525/ using document root <none> and password file <none>
I20260927 05:00:17.609608 30580 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:17.609737 30580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:17.610070 30580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:17.612960 30580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "5265f09e22d744529a0c9cc0a1969cfe"
format_stamp: "Formatted at 2026-09-27 05:00:17 on dist-test-slave-39f2"
server_key: "6a1a61c30d76f4062bc7604ff5cc567e"
server_key_iv: "4ba025e05554c34f89c554601c5f9b8b"
server_key_version: "encryptionkey@0"
I20260927 05:00:17.613737 30580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "5265f09e22d744529a0c9cc0a1969cfe"
format_stamp: "Formatted at 2026-09-27 05:00:17 on dist-test-slave-39f2"
server_key: "6a1a61c30d76f4062bc7604ff5cc567e"
server_key_iv: "4ba025e05554c34f89c554601c5f9b8b"
server_key_version: "encryptionkey@0"
I20260927 05:00:17.618781 30580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 05:00:17.622149 30595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:17.623549 30580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 05:00:17.623729 30580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "5265f09e22d744529a0c9cc0a1969cfe"
format_stamp: "Formatted at 2026-09-27 05:00:17 on dist-test-slave-39f2"
server_key: "6a1a61c30d76f4062bc7604ff5cc567e"
server_key_iv: "4ba025e05554c34f89c554601c5f9b8b"
server_key_version: "encryptionkey@0"
I20260927 05:00:17.623939 30580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:17.637020 30580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:17.637878 30580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:17.638175 30580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:17.656664 30580 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35885
I20260927 05:00:17.656749 30646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35885 every 8 connection(s)
I20260927 05:00:17.658063 30580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:00:17.661854 30647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:00:17.664225 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30580
I20260927 05:00:17.664443 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:00:17.665266 21157 external_mini_cluster.cc:1428] Setting key 40304be9275cde2c01ed4a65dfe67c54
I20260927 05:00:17.676167 30647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe: Bootstrap starting.
I20260927 05:00:17.680294 30647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe: Neither blocks nor log segments found. Creating new log.
I20260927 05:00:17.681736 30647 log.cc:824] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:17.681910 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:17.667472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.20.169.126:35885)
0927 05:00:17.668639 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:17.668662 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:17.668713 (+ 51us) server_negotiation.cc:407] Connection header received
0927 05:00:17.671051 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:17.671080 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:17.671672 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:17.672158 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:17.672745 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:17.674854 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:17.676195 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:17.677216 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:17.677407 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:17.678733 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:17.678742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:17.678760 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:17.679558 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:17.679677 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:17.679705 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:17.679806 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:17.680202 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 05:00:17.680550 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":331,"threads_started":1}
I20260927 05:00:17.685158 30647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe: No bootstrap required, opened a new log
I20260927 05:00:17.692207 30647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265f09e22d744529a0c9cc0a1969cfe" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35885 } }
I20260927 05:00:17.692567 30647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:00:17.692673 30647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5265f09e22d744529a0c9cc0a1969cfe, State: Initialized, Role: FOLLOWER
I20260927 05:00:17.693212 30647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265f09e22d744529a0c9cc0a1969cfe" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35885 } }
I20260927 05:00:17.693456 30647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:00:17.693595 30647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:00:17.693778 30647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:00:17.696091 30647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265f09e22d744529a0c9cc0a1969cfe" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35885 } }
I20260927 05:00:17.696561 30647 leader_election.cc:302] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5265f09e22d744529a0c9cc0a1969cfe; no voters:
I20260927 05:00:17.697208 30647 leader_election.cc:288] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:00:17.697418 30652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:00:17.698400 30652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [term 1 LEADER]: Becoming Leader. State: Replica: 5265f09e22d744529a0c9cc0a1969cfe, State: Running, Role: LEADER
I20260927 05:00:17.698881 30652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265f09e22d744529a0c9cc0a1969cfe" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35885 } }
I20260927 05:00:17.699331 30647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:00:17.701704 30654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5265f09e22d744529a0c9cc0a1969cfe. Latest consensus state: current_term: 1 leader_uuid: "5265f09e22d744529a0c9cc0a1969cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265f09e22d744529a0c9cc0a1969cfe" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35885 } } }
I20260927 05:00:17.701624 30653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5265f09e22d744529a0c9cc0a1969cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265f09e22d744529a0c9cc0a1969cfe" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35885 } } }
I20260927 05:00:17.702127 30654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:17.702152 30653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:17.705317 30660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:00:17.711038 30660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:00:17.720183 30660 catalog_manager.cc:1347] Generated new cluster ID: 14b6c806677f4760964a11a7dc52f2f8
I20260927 05:00:17.720311 30660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:00:17.743227 30660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:00:17.745095 30660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:00:17.754287 30660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5265f09e22d744529a0c9cc0a1969cfe: Generated new TSK 0
I20260927 05:00:17.755290 30660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:00:17.767583 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35885
--builtin_ntp_servers=127.20.169.84:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 05:00:18.104638 30671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:18.104959 30671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:18.105116 30671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:18.112519 30671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:18.112618 30671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:18.112787 30671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:00:18.121762 30671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:18.123059 30671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:18.124877 30671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:18.138200 30679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:18.138243 30677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:18.138155 30676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:18.139125 30671 server_base.cc:1034] running on GCE node
I20260927 05:00:18.139904 30671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:18.140793 30671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:18.142117 30671 hybrid_clock.cc:648] HybridClock initialized: now 1790485218142044 us; error 65 us; skew 500 ppm
I20260927 05:00:18.142439 30671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:18.148255 30671 webserver.cc:460] Webserver started at http://127.20.169.65:35763/ using document root <none> and password file <none>
I20260927 05:00:18.149055 30671 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:18.149190 30671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:18.149524 30671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:18.152446 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "0771e02513f14a1890feecc5db7404fc"
format_stamp: "Formatted at 2026-09-27 05:00:18 on dist-test-slave-39f2"
server_key: "3c46c475ed8a6ba00e4eb9dcb76635b9"
server_key_iv: "aa37339d6068d603f504e4ccfb722fa6"
server_key_version: "encryptionkey@0"
I20260927 05:00:18.153282 30671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "0771e02513f14a1890feecc5db7404fc"
format_stamp: "Formatted at 2026-09-27 05:00:18 on dist-test-slave-39f2"
server_key: "3c46c475ed8a6ba00e4eb9dcb76635b9"
server_key_iv: "aa37339d6068d603f504e4ccfb722fa6"
server_key_version: "encryptionkey@0"
I20260927 05:00:18.158533 30671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 05:00:18.162091 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:18.163518 30671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 05:00:18.163721 30671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "0771e02513f14a1890feecc5db7404fc"
format_stamp: "Formatted at 2026-09-27 05:00:18 on dist-test-slave-39f2"
server_key: "3c46c475ed8a6ba00e4eb9dcb76635b9"
server_key_iv: "aa37339d6068d603f504e4ccfb722fa6"
server_key_version: "encryptionkey@0"
I20260927 05:00:18.163954 30671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:18.175065 30671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:18.175848 30671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:18.176146 30671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:18.177100 30671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:18.178447 30671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:18.178567 30671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:18.178701 30671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:18.178808 30671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:18.211897 30671 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:46525
I20260927 05:00:18.211984 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:46525 every 8 connection(s)
I20260927 05:00:18.213361 30671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:00:18.221285 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30671
I20260927 05:00:18.221493 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:00:18.222231 21157 external_mini_cluster.cc:1428] Setting key 166cee5fc7a0418a246493f69d4c1f93
I20260927 05:00:18.225231 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35885
--builtin_ntp_servers=127.20.169.84:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 05:00:18.226557 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:18.216023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:54377 (local address 127.20.169.126:35885)
0927 05:00:18.216484 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:18.216489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:18.217755 (+ 1266us) server_negotiation.cc:407] Connection header received
0927 05:00:18.219430 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:18.219434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:18.219497 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:18.219595 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:18.220635 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.221689 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.223523 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.224036 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.224124 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:18.225568 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:18.225576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:18.225583 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:18.225606 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:18.225665 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:18.225683 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:18.225775 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:18.226004 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 05:00:18.226108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260927 05:00:18.227326 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:18.216655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35885 (local address 127.20.169.65:54377)
0927 05:00:18.217595 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:18.217634 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:18.219186 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:18.219788 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:18.219796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:18.219909 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:18.220458 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:18.220474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.221849 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.221854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:18.223391 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:18.223404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.224272 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.224279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:18.224501 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:18.225234 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:00:18.225237 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:18.225277 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:00:18.225377 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:18.225778 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:18.225786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:18.225807 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 05:00:18.225966 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 05:00:18.226251 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":314,"threads_started":1}
I20260927 05:00:18.228825 30799 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35885
I20260927 05:00:18.229126 30799 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:18.230027 30799 heartbeater.cc:511] Master 127.20.169.126:35885 requested a full tablet report, sending...
I20260927 05:00:18.232105 30612 ts_manager.cc:194] Registered new tserver with Master: 0771e02513f14a1890feecc5db7404fc (127.20.169.65:46525)
I20260927 05:00:18.233703 30612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.65:54377
W20260927 05:00:18.551128 30803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:18.551401 30803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:18.551501 30803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:18.558717 30803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:18.558812 30803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:18.558918 30803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:00:18.568063 30803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:18.569607 30803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:18.571473 30803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:18.584559 30809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:18.584599 30808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:18.584627 30812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:18.585740 30803 server_base.cc:1034] running on GCE node
I20260927 05:00:18.586282 30803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:18.587141 30803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:18.588383 30803 hybrid_clock.cc:648] HybridClock initialized: now 1790485218588354 us; error 37 us; skew 500 ppm
I20260927 05:00:18.588699 30803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:18.591053 30803 webserver.cc:460] Webserver started at http://127.20.169.66:45243/ using document root <none> and password file <none>
I20260927 05:00:18.591732 30803 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:18.591873 30803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:18.592196 30803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:18.595000 30803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "00951aa40e8d4db1b5aa6aa8067de259"
format_stamp: "Formatted at 2026-09-27 05:00:18 on dist-test-slave-39f2"
server_key: "97578ec36158140f2829bd550d1a52a1"
server_key_iv: "955318d907b664af73cdb8208ccff650"
server_key_version: "encryptionkey@0"
I20260927 05:00:18.595768 30803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "00951aa40e8d4db1b5aa6aa8067de259"
format_stamp: "Formatted at 2026-09-27 05:00:18 on dist-test-slave-39f2"
server_key: "97578ec36158140f2829bd550d1a52a1"
server_key_iv: "955318d907b664af73cdb8208ccff650"
server_key_version: "encryptionkey@0"
I20260927 05:00:18.601244 30803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 05:00:18.604588 30818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:18.606060 30803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:00:18.606226 30803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "00951aa40e8d4db1b5aa6aa8067de259"
format_stamp: "Formatted at 2026-09-27 05:00:18 on dist-test-slave-39f2"
server_key: "97578ec36158140f2829bd550d1a52a1"
server_key_iv: "955318d907b664af73cdb8208ccff650"
server_key_version: "encryptionkey@0"
I20260927 05:00:18.606439 30803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:18.620275 30803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:18.621249 30803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:18.621672 30803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:18.622709 30803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:18.624073 30803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:18.624158 30803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:18.624230 30803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:18.624285 30803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:18.657455 30803 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45669
I20260927 05:00:18.657557 30931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45669 every 8 connection(s)
I20260927 05:00:18.658766 30803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:00:18.660416 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30803
I20260927 05:00:18.660614 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:00:18.661114 21157 external_mini_cluster.cc:1428] Setting key bd7da4e94b723e250203977f2730788b
I20260927 05:00:18.664206 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35885
--builtin_ntp_servers=127.20.169.84:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 05:00:18.672112 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:18.661631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:44847 (local address 127.20.169.126:35885)
0927 05:00:18.661829 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:18.661835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:18.663640 (+ 1805us) server_negotiation.cc:407] Connection header received
0927 05:00:18.665531 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:18.665536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:18.665624 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:18.665728 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:18.666875 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.667624 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.669411 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.669822 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.669954 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:18.671328 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:18.671333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:18.671336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:18.671352 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:18.671382 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:18.671394 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:18.671488 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:18.671710 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 05:00:18.671803 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:00:18.673594 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:18.662274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35885 (local address 127.20.169.66:44847)
0927 05:00:18.663466 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:18.663521 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:18.665267 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:18.666053 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:18.666061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:18.666148 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:18.666707 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:18.666722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.667781 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.667786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:18.669248 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:18.669260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:18.670048 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:18.670055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:18.670277 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:18.671009 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:00:18.671012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:18.671050 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:00:18.671124 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:18.671488 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:18.671501 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:18.671512 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:00:18.671700 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 05:00:18.672086 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":336,"threads_started":1}
I20260927 05:00:18.675032 30932 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35885
I20260927 05:00:18.675379 30932 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:18.676126 30932 heartbeater.cc:511] Master 127.20.169.126:35885 requested a full tablet report, sending...
I20260927 05:00:18.677525 30612 ts_manager.cc:194] Registered new tserver with Master: 00951aa40e8d4db1b5aa6aa8067de259 (127.20.169.66:45669)
I20260927 05:00:18.678341 30612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.66:44847
W20260927 05:00:18.994872 30935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:18.995126 30935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:18.995271 30935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:19.002601 30935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:19.002693 30935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:19.002858 30935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:00:19.012193 30935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:19.013482 30935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:19.015228 30935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:19.028618 30941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:19.028656 30943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:19.028618 30940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:19.029814 30935 server_base.cc:1034] running on GCE node
I20260927 05:00:19.030378 30935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:19.031278 30935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:19.032521 30935 hybrid_clock.cc:648] HybridClock initialized: now 1790485219032469 us; error 46 us; skew 500 ppm
I20260927 05:00:19.032895 30935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:19.035229 30935 webserver.cc:460] Webserver started at http://127.20.169.67:43625/ using document root <none> and password file <none>
I20260927 05:00:19.035928 30935 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:19.036062 30935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:19.036386 30935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:19.039330 30935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "a58dde04ab8540c9826c54378c404dab"
format_stamp: "Formatted at 2026-09-27 05:00:19 on dist-test-slave-39f2"
server_key: "c2d9f9f46f2ea1db16417551d285161e"
server_key_iv: "09ad68920505f72e1d91ac1da2eb3279"
server_key_version: "encryptionkey@0"
I20260927 05:00:19.040113 30935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "a58dde04ab8540c9826c54378c404dab"
format_stamp: "Formatted at 2026-09-27 05:00:19 on dist-test-slave-39f2"
server_key: "c2d9f9f46f2ea1db16417551d285161e"
server_key_iv: "09ad68920505f72e1d91ac1da2eb3279"
server_key_version: "encryptionkey@0"
I20260927 05:00:19.045341 30935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 05:00:19.048780 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:19.050232 30935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:00:19.050408 30935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "a58dde04ab8540c9826c54378c404dab"
format_stamp: "Formatted at 2026-09-27 05:00:19 on dist-test-slave-39f2"
server_key: "c2d9f9f46f2ea1db16417551d285161e"
server_key_iv: "09ad68920505f72e1d91ac1da2eb3279"
server_key_version: "encryptionkey@0"
I20260927 05:00:19.050633 30935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:19.069561 30935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:19.070382 30935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:19.070685 30935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:19.071609 30935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:19.073050 30935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:19.073171 30935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:19.073303 30935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:19.073391 30935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:19.107056 30935 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:37041
I20260927 05:00:19.107138 31062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:37041 every 8 connection(s)
I20260927 05:00:19.108366 30935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:00:19.115767 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 30935
I20260927 05:00:19.115959 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:00:19.116393 21157 external_mini_cluster.cc:1428] Setting key e8f3d3de45048bf13c6b5f7bf8af3c34
I20260927 05:00:19.121415 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:19.111159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:50801 (local address 127.20.169.126:35885)
0927 05:00:19.111337 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:19.111342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:19.112716 (+ 1374us) server_negotiation.cc:407] Connection header received
0927 05:00:19.114426 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:19.114430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:19.114497 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:19.114583 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:19.115695 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.116569 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.118688 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.119344 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.119457 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:19.120754 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:19.120758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:19.120761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:19.120776 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:19.120800 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:19.120856 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:19.120935 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:19.121114 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 05:00:19.121184 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:00:19.122393 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:19.111707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35885 (local address 127.20.169.67:50801)
0927 05:00:19.112563 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:19.112604 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:19.114190 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:19.114777 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:19.114785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:19.114865 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:19.115519 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:19.115547 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.116715 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.116720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:19.118561 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:19.118572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.119594 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.119599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:19.119786 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:19.120488 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:00:19.120490 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:19.120527 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:00:19.120602 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:19.120937 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:19.120945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:19.120950 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:00:19.121091 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 05:00:19.121365 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260927 05:00:19.123670 31063 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35885
I20260927 05:00:19.123906 31063 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:19.124579 31063 heartbeater.cc:511] Master 127.20.169.126:35885 requested a full tablet report, sending...
I20260927 05:00:19.125893 30612 ts_manager.cc:194] Registered new tserver with Master: a58dde04ab8540c9826c54378c404dab (127.20.169.67:37041)
I20260927 05:00:19.126647 30612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.67:50801
I20260927 05:00:19.131476 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:00:19.137602 30801 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:19.136083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.20.169.126:35885)
0927 05:00:19.136326 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:19.136335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:19.136665 (+ 330us) server_negotiation.cc:407] Connection header received
0927 05:00:19.136774 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:19.136777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:19.136904 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:19.136988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:19.137389 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50480: BlockingRecv error: recv got EOF from 127.0.0.1:50480 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:00:19.138010 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30671
I20260927 05:00:19.145730 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:19.138948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.20.169.65:46525)
0927 05:00:19.139441 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:19.139451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:19.139482 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:00:19.139534 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:19.139540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:19.139723 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:19.139843 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:19.140273 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.141325 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.142657 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.143329 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.143425 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:19.144402 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:19.144410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:19.144421 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:19.145030 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:19.145073 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:19.145092 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:19.145183 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:19.145314 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 05:00:19.145461 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":255,"threads_started":1}
I20260927 05:00:19.146962 30794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:19.229809 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30671
I20260927 05:00:19.245076 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30803
I20260927 05:00:19.253746 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:19.246032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.20.169.66:45669)
0927 05:00:19.246604 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:19.246611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:19.246639 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:00:19.246709 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:19.246717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:19.246963 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:19.247151 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:19.247594 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.248694 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.250118 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.250731 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.250888 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:19.252007 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:19.252016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:19.252030 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:19.252706 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:19.252766 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:19.252795 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:19.253046 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:19.253216 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 05:00:19.253386 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":313,"threads_started":1}
I20260927 05:00:19.255185 30927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:19.337724 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30803
I20260927 05:00:19.355398 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30935
I20260927 05:00:19.363914 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:19.356549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.20.169.67:37041)
0927 05:00:19.357221 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:19.357228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:19.357257 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:00:19.357343 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:19.357349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:19.357576 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:19.357696 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:19.358147 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.359185 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.360518 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:19.361149 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:19.361244 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:19.362413 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:19.362423 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:19.362436 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:19.363045 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:19.363091 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:19.363116 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:19.363206 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:19.363400 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 05:00:19.363567 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":396,"threads_started":1}
I20260927 05:00:19.365228 31058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:19.453136 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30935
I20260927 05:00:19.471563 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 30580
I20260927 05:00:19.472867 30642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:19.573904 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 30580
2026-09-27T05:00:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2418 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T05:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:00:19Z Disabled control of system clock
I20260927 05:00:19.648298 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35151
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 05:00:19.977937 31082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:19.978313 31082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:19.978492 31082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:19.985654 31082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:00:19.985802 31082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:19.985901 31082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:19.985993 31082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:00:19.986060 31082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:00:19.994990 31082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:19.996316 31082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:19.998123 31082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:20.010085 31087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:20.010156 31090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:20.010134 31088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:20.011070 31082 server_base.cc:1034] running on GCE node
I20260927 05:00:20.011830 31082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:20.013216 31082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:20.014441 31082 hybrid_clock.cc:648] HybridClock initialized: now 1790485220014385 us; error 42 us; skew 500 ppm
I20260927 05:00:20.014829 31082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:20.017400 31082 webserver.cc:460] Webserver started at http://127.20.169.126:45999/ using document root <none> and password file <none>
I20260927 05:00:20.018100 31082 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:20.018214 31082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:20.018564 31082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:20.023315 31082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "6a01fab9074f4736b442e27ffd24a56f"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "a0a6bfdd08102adc0c999ca7f55abd4e"
server_key_iv: "96f3d3ecb4c8d5ee12bb7aefaa97b391"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.024106 31082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "6a01fab9074f4736b442e27ffd24a56f"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "a0a6bfdd08102adc0c999ca7f55abd4e"
server_key_iv: "96f3d3ecb4c8d5ee12bb7aefaa97b391"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.029377 31082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:00:20.032925 31096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:20.034263 31082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:00:20.034497 31082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "6a01fab9074f4736b442e27ffd24a56f"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "a0a6bfdd08102adc0c999ca7f55abd4e"
server_key_iv: "96f3d3ecb4c8d5ee12bb7aefaa97b391"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.034724 31082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:20.040503 31082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:20.041296 31082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:20.041591 31082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:20.060103 31082 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35151
I20260927 05:00:20.060184 31148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35151 every 8 connection(s)
I20260927 05:00:20.061445 31082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:00:20.065250 31149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:00:20.070246 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31082
I20260927 05:00:20.070472 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:00:20.071128 21157 external_mini_cluster.cc:1428] Setting key 8a8c95f7223a00f626b3b68ddf709764
I20260927 05:00:20.078182 31149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f: Bootstrap starting.
I20260927 05:00:20.081437 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:20.073385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.20.169.126:35151)
0927 05:00:20.074587 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:20.074604 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:20.074658 (+ 54us) server_negotiation.cc:407] Connection header received
0927 05:00:20.077205 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:20.077234 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:20.077772 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:20.078238 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:20.079298 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:20.079307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:20.079323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:20.079348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:20.079423 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:20.079447 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:20.079675 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:20.079814 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 05:00:20.080137 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":351,"threads_started":1}
I20260927 05:00:20.082355 31149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f: Neither blocks nor log segments found. Creating new log.
I20260927 05:00:20.083488 31149 log.cc:824] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:20.086473 31149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f: No bootstrap required, opened a new log
I20260927 05:00:20.094388 31149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a01fab9074f4736b442e27ffd24a56f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35151 } }
I20260927 05:00:20.094774 31149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:00:20.094900 31149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a01fab9074f4736b442e27ffd24a56f, State: Initialized, Role: FOLLOWER
I20260927 05:00:20.095474 31149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a01fab9074f4736b442e27ffd24a56f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35151 } }
I20260927 05:00:20.095803 31149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:00:20.095990 31149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:00:20.096207 31149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:00:20.098563 31149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a01fab9074f4736b442e27ffd24a56f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35151 } }
I20260927 05:00:20.099097 31149 leader_election.cc:302] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a01fab9074f4736b442e27ffd24a56f; no voters:
I20260927 05:00:20.099694 31149 leader_election.cc:288] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:00:20.099808 31154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:00:20.100571 31154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [term 1 LEADER]: Becoming Leader. State: Replica: 6a01fab9074f4736b442e27ffd24a56f, State: Running, Role: LEADER
I20260927 05:00:20.101173 31154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a01fab9074f4736b442e27ffd24a56f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35151 } }
I20260927 05:00:20.102036 31149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:00:20.103785 31156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a01fab9074f4736b442e27ffd24a56f. Latest consensus state: current_term: 1 leader_uuid: "6a01fab9074f4736b442e27ffd24a56f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a01fab9074f4736b442e27ffd24a56f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35151 } } }
I20260927 05:00:20.104172 31156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:20.104079 31155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a01fab9074f4736b442e27ffd24a56f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a01fab9074f4736b442e27ffd24a56f" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35151 } } }
I20260927 05:00:20.104780 31155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:20.107683 31159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:00:20.114872 31159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:00:20.124464 31159 catalog_manager.cc:1347] Generated new cluster ID: ff032a5240a1445cb29c774f389b596d
I20260927 05:00:20.124604 31159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:00:20.136394 31159 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:00:20.136785 31159 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:00:20.149369 31159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a01fab9074f4736b442e27ffd24a56f: Generated new TSK 0
I20260927 05:00:20.150164 31159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:00:20.157527 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35151
--builtin_ntp_servers=127.20.169.84:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 05:00:20.488634 31173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:20.488972 31173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:20.489141 31173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:20.496500 31173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:20.496590 31173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:20.496743 31173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:00:20.506176 31173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:20.507421 31173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:20.509182 31173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:20.522425 31178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:20.522377 31181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:20.522460 31179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:20.523736 31173 server_base.cc:1034] running on GCE node
I20260927 05:00:20.524348 31173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:20.525296 31173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:20.526525 31173 hybrid_clock.cc:648] HybridClock initialized: now 1790485220526471 us; error 26 us; skew 500 ppm
I20260927 05:00:20.526896 31173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:20.529477 31173 webserver.cc:460] Webserver started at http://127.20.169.65:38067/ using document root <none> and password file <none>
I20260927 05:00:20.530121 31173 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:20.530207 31173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:20.530491 31173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:20.533371 31173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "5c7e71ba8b6449ba82b7e6fdb360c2dc"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "b8b102494ad3e0c2de45624e63088527"
server_key_iv: "a7ac94063f86eb9587cc586a796e5b65"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.534085 31173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "5c7e71ba8b6449ba82b7e6fdb360c2dc"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "b8b102494ad3e0c2de45624e63088527"
server_key_iv: "a7ac94063f86eb9587cc586a796e5b65"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.538996 31173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:00:20.542424 31187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:20.543824 31173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:00:20.544000 31173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "5c7e71ba8b6449ba82b7e6fdb360c2dc"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "b8b102494ad3e0c2de45624e63088527"
server_key_iv: "a7ac94063f86eb9587cc586a796e5b65"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.544243 31173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:20.549842 31173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:20.550606 31173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:20.551028 31173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:20.551997 31173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:20.553436 31173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:20.553525 31173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:20.553679 31173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:20.553784 31173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:20.587059 31173 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:34677
I20260927 05:00:20.587134 31300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:34677 every 8 connection(s)
I20260927 05:00:20.588393 31173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:00:20.592136 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31173
I20260927 05:00:20.592329 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:00:20.593044 21157 external_mini_cluster.cc:1428] Setting key 929b286360f9cae8f46f48644922af0d
I20260927 05:00:20.596083 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35151
--builtin_ntp_servers=127.20.169.84:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 05:00:20.597129 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:20.591109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:57967 (local address 127.20.169.126:35151)
0927 05:00:20.591562 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:20.591567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:20.592802 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 05:00:20.594724 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:20.594730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:20.594800 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:20.594927 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:20.596407 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:20.596412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:20.596415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:20.596430 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:20.596473 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:20.596484 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:20.596581 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:20.596805 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 05:00:20.596915 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":255,"threads_started":1}
I20260927 05:00:20.598405 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:20.591586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35151 (local address 127.20.169.65:57967)
0927 05:00:20.592622 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:20.592673 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:20.594427 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:20.595218 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:20.595229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:20.595349 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:20.596054 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:00:20.596058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:20.596111 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:00:20.596227 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:20.596587 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:20.596600 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:20.596610 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:00:20.596936 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 05:00:20.597194 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":335,"threads_started":1}
I20260927 05:00:20.599903 31301 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35151
I20260927 05:00:20.600194 31301 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:20.600796 31301 heartbeater.cc:511] Master 127.20.169.126:35151 requested a full tablet report, sending...
I20260927 05:00:20.603441 31113 ts_manager.cc:194] Registered new tserver with Master: 5c7e71ba8b6449ba82b7e6fdb360c2dc (127.20.169.65:34677)
W20260927 05:00:20.925630 31305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:20.925884 31305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:20.926024 31305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:20.933226 31305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:20.933310 31305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:20.933481 31305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:00:20.942584 31305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:20.943869 31305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:20.945665 31305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:20.959244 31311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:20.959383 31310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:20.959273 31313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:20.960111 31305 server_base.cc:1034] running on GCE node
I20260927 05:00:20.960747 31305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:20.961700 31305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:20.962930 31305 hybrid_clock.cc:648] HybridClock initialized: now 1790485220962874 us; error 43 us; skew 500 ppm
I20260927 05:00:20.963235 31305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:20.965525 31305 webserver.cc:460] Webserver started at http://127.20.169.66:44465/ using document root <none> and password file <none>
I20260927 05:00:20.966207 31305 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:20.966355 31305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:20.966677 31305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:20.969538 31305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "2997ef56110b403f87305f43a1bc2cc3"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "53b308526f4500faf5be0ef29cf635e6"
server_key_iv: "72ee056fdb4436520dafd1128a3ba5a5"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.970332 31305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "2997ef56110b403f87305f43a1bc2cc3"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "53b308526f4500faf5be0ef29cf635e6"
server_key_iv: "72ee056fdb4436520dafd1128a3ba5a5"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.975226 31305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:00:20.978866 31319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:20.980379 31305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:00:20.980561 31305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "2997ef56110b403f87305f43a1bc2cc3"
format_stamp: "Formatted at 2026-09-27 05:00:20 on dist-test-slave-39f2"
server_key: "53b308526f4500faf5be0ef29cf635e6"
server_key_iv: "72ee056fdb4436520dafd1128a3ba5a5"
server_key_version: "encryptionkey@0"
I20260927 05:00:20.980790 31305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:20.986476 31305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:20.987267 31305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:20.987564 31305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:20.988466 31305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:20.990036 31305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:20.990161 31305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:20.990294 31305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:20.990388 31305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:21.022804 31305 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:46691
I20260927 05:00:21.022892 31432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:46691 every 8 connection(s)
I20260927 05:00:21.024123 31305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:00:21.030843 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31305
I20260927 05:00:21.031029 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:00:21.031527 21157 external_mini_cluster.cc:1428] Setting key 79992278456f2ad0df9424d8b6dc1fcc
I20260927 05:00:21.033002 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.026884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:38365 (local address 127.20.169.126:35151)
0927 05:00:21.027075 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:21.027080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:21.028472 (+ 1392us) server_negotiation.cc:407] Connection header received
0927 05:00:21.030484 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:21.030491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:21.030586 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:21.030716 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:21.032224 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:21.032231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:21.032236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:21.032257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:21.032300 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:21.032316 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:21.032471 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:21.032647 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 05:00:21.032715 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:00:21.033996 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.027332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35151 (local address 127.20.169.66:38365)
0927 05:00:21.028298 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:21.028336 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:21.030152 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:21.031004 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:21.031017 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:21.031157 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:21.031919 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:00:21.031922 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:21.031960 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:00:21.032051 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:21.032419 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:21.032430 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:21.032442 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:00:21.032745 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 05:00:21.033043 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260927 05:00:21.034355 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35151
--builtin_ntp_servers=127.20.169.84:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 05:00:21.035353 31433 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35151
I20260927 05:00:21.035696 31433 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:21.036303 31433 heartbeater.cc:511] Master 127.20.169.126:35151 requested a full tablet report, sending...
I20260927 05:00:21.037958 31113 ts_manager.cc:194] Registered new tserver with Master: 2997ef56110b403f87305f43a1bc2cc3 (127.20.169.66:46691)
W20260927 05:00:21.374444 31436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:21.374701 31436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:21.374841 31436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:21.381982 31436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:21.382073 31436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:21.382238 31436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:00:21.391428 31436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:21.392701 31436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:21.394447 31436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:21.407390 31441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:21.407361 31442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:21.407832 31444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:21.408562 31436 server_base.cc:1034] running on GCE node
I20260927 05:00:21.409255 31436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:21.410211 31436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:21.411438 31436 hybrid_clock.cc:648] HybridClock initialized: now 1790485221411400 us; error 43 us; skew 500 ppm
I20260927 05:00:21.411747 31436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:00:21.414012 31436 webserver.cc:460] Webserver started at http://127.20.169.67:39777/ using document root <none> and password file <none>
I20260927 05:00:21.414667 31436 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:21.414778 31436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:21.415105 31436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:21.417979 31436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "c7528b522ed34a95bab99fb4e7e0912e"
format_stamp: "Formatted at 2026-09-27 05:00:21 on dist-test-slave-39f2"
server_key: "be1c55a3f7c3b0ecaf4bc94687bcb40c"
server_key_iv: "d226ca93fe83b492c0a6814026e234fe"
server_key_version: "encryptionkey@0"
I20260927 05:00:21.418769 31436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "c7528b522ed34a95bab99fb4e7e0912e"
format_stamp: "Formatted at 2026-09-27 05:00:21 on dist-test-slave-39f2"
server_key: "be1c55a3f7c3b0ecaf4bc94687bcb40c"
server_key_iv: "d226ca93fe83b492c0a6814026e234fe"
server_key_version: "encryptionkey@0"
I20260927 05:00:21.423632 31436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:00:21.426913 31450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:21.428205 31436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:00:21.428413 31436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "c7528b522ed34a95bab99fb4e7e0912e"
format_stamp: "Formatted at 2026-09-27 05:00:21 on dist-test-slave-39f2"
server_key: "be1c55a3f7c3b0ecaf4bc94687bcb40c"
server_key_iv: "d226ca93fe83b492c0a6814026e234fe"
server_key_version: "encryptionkey@0"
I20260927 05:00:21.428630 31436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:21.434000 31436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:21.434775 31436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:21.435058 31436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:21.435968 31436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:21.437541 31436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:21.437696 31436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:21.437834 31436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:21.437924 31436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:21.471019 31436 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:38749
I20260927 05:00:21.471108 31562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:38749 every 8 connection(s)
I20260927 05:00:21.472368 31436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:00:21.478392 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31436
I20260927 05:00:21.478577 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:00:21.479014 21157 external_mini_cluster.cc:1428] Setting key 94367f89dde99ac68561e36cad969e26
I20260927 05:00:21.481773 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.475247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:34207 (local address 127.20.169.126:35151)
0927 05:00:21.475527 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:21.475531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:21.477077 (+ 1546us) server_negotiation.cc:407] Connection header received
0927 05:00:21.478956 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:21.478962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:21.479042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:21.479158 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:21.480953 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:21.480961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:21.480970 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:21.481009 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:21.481056 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:21.481073 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:21.481200 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:21.481493 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 05:00:21.481551 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 05:00:21.482865 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.475829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35151 (local address 127.20.169.67:34207)
0927 05:00:21.476897 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:21.476939 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:21.478633 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:21.479411 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:21.479420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:21.479561 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:21.480524 (+ 963us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:00:21.480527 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:21.480583 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:00:21.480710 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:21.481213 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:21.481225 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:21.481233 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 05:00:21.481593 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 05:00:21.481881 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":297,"threads_started":1}
I20260927 05:00:21.484148 31563 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35151
I20260927 05:00:21.484489 31563 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:21.485103 31563 heartbeater.cc:511] Master 127.20.169.126:35151 requested a full tablet report, sending...
I20260927 05:00:21.486392 31113 ts_manager.cc:194] Registered new tserver with Master: c7528b522ed34a95bab99fb4e7e0912e (127.20.169.67:38749)
I20260927 05:00:21.494494 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:00:21.500533 31303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:00:21.498887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.20.169.126:35151)
0927 05:00:21.499099 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:21.499107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:21.499535 (+ 428us) server_negotiation.cc:407] Connection header received
0927 05:00:21.499730 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:21.499735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:21.499797 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:21.499906 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:21.500261 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46666: BlockingRecv error: recv got EOF from 127.0.0.1:46666 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:00:21.501518 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31173
I20260927 05:00:21.505098 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.502411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.20.169.65:34677)
0927 05:00:21.502945 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:21.502952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:21.502979 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:00:21.503053 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:21.503059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:21.503269 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:21.503398 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:21.504379 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:21.504387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:21.504399 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:21.504412 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:21.504462 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:21.504477 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:21.504580 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:21.504732 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 05:00:21.504884 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260927 05:00:21.506395 31296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:21.588078 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31173
I20260927 05:00:21.605568 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31305
I20260927 05:00:21.609372 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.606619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.20.169.66:46691)
0927 05:00:21.607155 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:21.607162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:21.607197 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:00:21.607257 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:21.607263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:21.607484 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:21.607611 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:21.608560 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:21.608567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:21.608580 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:21.608593 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:21.608639 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:21.608654 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:21.608759 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:21.608956 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 05:00:21.609074 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260927 05:00:21.610718 31428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:21.692664 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31305
I20260927 05:00:21.710135 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31436
I20260927 05:00:21.714213 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:21.711225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.20.169.67:38749)
0927 05:00:21.711871 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:21.711879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:21.711910 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:00:21.711971 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:21.711977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:21.712191 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:21.712316 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:21.713392 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:21.713399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:21.713411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:00:21.713427 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:21.713468 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:00:21.713481 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:21.713613 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:21.713802 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 05:00:21.713899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":307,"threads_started":1}
I20260927 05:00:21.715597 31558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:21.799407 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31436
I20260927 05:00:21.818856 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31082
I20260927 05:00:21.820134 31144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:21.924178 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31082
2026-09-27T05:00:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2349 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:00:22 dist-test-slave-39f2 krb5kdc[31581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:00:22 dist-test-slave-39f2 krb5kdc[31581](info): set up 2 sockets
Sep 27 05:00:22 dist-test-slave-39f2 krb5kdc[31581](info): commencing operation
krb5kdc: starting...
W20260927 05:00:23.033829 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:00:23 dist-test-slave-39f2 krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:00:23.269629 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:34611
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:00:23.605497 31594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:23.605813 31594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:23.606001 31594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:23.613680 31594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:00:23.613799 31594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:23.613921 31594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:23.614035 31594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:00:23.614135 31594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:00:23.624256 31594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:23.625731 31594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:23.627068 31594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:23.639086 31599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:23.639108 31602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:23.639010 31600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:23.639990 31594 server_base.cc:1034] running on GCE node
I20260927 05:00:23.640786 31594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:23.642268 31594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:23.643469 31594 hybrid_clock.cc:648] HybridClock initialized: now 1790485223643401 us; error 63 us; skew 500 ppm
I20260927 05:00:23.643867 31594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:23 dist-test-slave-39f2 krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:23.653604 31594 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:00:23.655134 31594 webserver.cc:460] Webserver started at http://127.20.169.126:39091/ using document root <none> and password file <none>
I20260927 05:00:23.655790 31594 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:23.655891 31594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:23.656132 31594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:23.659075 31594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "447403f7849a463aa60fdbe5fab99f65"
format_stamp: "Formatted at 2026-09-27 05:00:23 on dist-test-slave-39f2"
server_key: "f1e23645f5343c59b27529e708ce5787"
server_key_iv: "e8a0a53145869fad8e3a0340ba5076a0"
server_key_version: "encryptionkey@0"
I20260927 05:00:23.659754 31594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "447403f7849a463aa60fdbe5fab99f65"
format_stamp: "Formatted at 2026-09-27 05:00:23 on dist-test-slave-39f2"
server_key: "f1e23645f5343c59b27529e708ce5787"
server_key_iv: "e8a0a53145869fad8e3a0340ba5076a0"
server_key_version: "encryptionkey@0"
I20260927 05:00:23.665210 31594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 05:00:23.669111 31609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:23.670672 31594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:00:23.670876 31594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "447403f7849a463aa60fdbe5fab99f65"
format_stamp: "Formatted at 2026-09-27 05:00:23 on dist-test-slave-39f2"
server_key: "f1e23645f5343c59b27529e708ce5787"
server_key_iv: "e8a0a53145869fad8e3a0340ba5076a0"
server_key_version: "encryptionkey@0"
I20260927 05:00:23.671087 31594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:23.684309 31594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:23.689195 31594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:23.689550 31594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:23.708592 31594 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:34611
I20260927 05:00:23.708630 31662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:34611 every 8 connection(s)
I20260927 05:00:23.710004 31594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:00:23.712577 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31594
I20260927 05:00:23.712874 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:00:23.713541 21157 external_mini_cluster.cc:1428] Setting key dbc81c6fdf1e1673985f03cd22e47dad
I20260927 05:00:23.714138 31663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:00:23.729231 31663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65: Bootstrap starting.
Sep 27 05:00:23 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:00:23.733536 31663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65: Neither blocks nor log segments found. Creating new log.
I20260927 05:00:23.734956 31663 log.cc:824] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:23.738615 31663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65: No bootstrap required, opened a new log
I20260927 05:00:23.739773 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:23.715529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.20.169.126:34611)
0927 05:00:23.716298 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:23.716310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:23.716356 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:00:23.718735 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:23.718756 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:23.719137 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:23.719518 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:23.721510 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:23.722867 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:23.724766 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:23.725579 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:23.725729 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:23.730616 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:23.730627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:23.730646 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:23.730704 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:23.734829 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:23.735683 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:23.735695 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:23.735703 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:23.735893 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:23.736377 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:23.736382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:23.736386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:23.737165 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:23.737478 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:23.737828 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 05:00:23.738252 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":348,"threads_started":1}
I20260927 05:00:23.748466 31663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447403f7849a463aa60fdbe5fab99f65" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34611 } }
I20260927 05:00:23.748922 31663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:00:23.749055 31663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447403f7849a463aa60fdbe5fab99f65, State: Initialized, Role: FOLLOWER
I20260927 05:00:23.749666 31663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447403f7849a463aa60fdbe5fab99f65" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34611 } }
I20260927 05:00:23.749915 31663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:00:23.750064 31663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:00:23.750273 31663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:00:23.752563 31663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447403f7849a463aa60fdbe5fab99f65" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34611 } }
I20260927 05:00:23.753129 31663 leader_election.cc:302] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 447403f7849a463aa60fdbe5fab99f65; no voters:
I20260927 05:00:23.753739 31663 leader_election.cc:288] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:00:23.753904 31668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:00:23.754627 31668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [term 1 LEADER]: Becoming Leader. State: Replica: 447403f7849a463aa60fdbe5fab99f65, State: Running, Role: LEADER
I20260927 05:00:23.755151 31668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447403f7849a463aa60fdbe5fab99f65" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34611 } }
I20260927 05:00:23.755926 31663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:00:23.758010 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 447403f7849a463aa60fdbe5fab99f65. Latest consensus state: current_term: 1 leader_uuid: "447403f7849a463aa60fdbe5fab99f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447403f7849a463aa60fdbe5fab99f65" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34611 } } }
I20260927 05:00:23.758396 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:23.758873 31669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "447403f7849a463aa60fdbe5fab99f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "447403f7849a463aa60fdbe5fab99f65" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34611 } } }
I20260927 05:00:23.759213 31669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65 [sys.catalog]: This master's current role is: LEADER
I20260927 05:00:23.761650 31675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:00:23.766932 31675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:00:23.778230 31675 catalog_manager.cc:1347] Generated new cluster ID: bdc308825ac6486a96de0721ab514b11
I20260927 05:00:23.778342 31675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:00:23.802702 31675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:00:23.804574 31675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:00:23.820502 31675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65: Generated new TSK 0
I20260927 05:00:23.821621 31675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:00:23.884084 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34611
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:00:24.211398 31689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:24.211683 31689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:24.211877 31689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:24.218853 31689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:24.218946 31689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:24.219137 31689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:00:24.228381 31689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:24.229630 31689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:24.230917 31689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:24.244082 31694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:24.244017 31697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:24.244059 31695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:24.244874 31689 server_base.cc:1034] running on GCE node
I20260927 05:00:24.245424 31689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:24.246349 31689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:24.247550 31689 hybrid_clock.cc:648] HybridClock initialized: now 1790485224247510 us; error 39 us; skew 500 ppm
I20260927 05:00:24.247867 31689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:24 dist-test-slave-39f2 krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:24.251189 31689 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:00:24.252596 31689 webserver.cc:460] Webserver started at http://127.20.169.65:46289/ using document root <none> and password file <none>
I20260927 05:00:24.253345 31689 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:24.253492 31689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:24.253790 31689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:24.256857 31689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "d92580abfbc845b988c398cc343cf77f"
format_stamp: "Formatted at 2026-09-27 05:00:24 on dist-test-slave-39f2"
server_key: "d331789871791fad44958ce9ca7c61b3"
server_key_iv: "fe313b95005b336dc3731402aaa5cbf8"
server_key_version: "encryptionkey@0"
I20260927 05:00:24.257637 31689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "d92580abfbc845b988c398cc343cf77f"
format_stamp: "Formatted at 2026-09-27 05:00:24 on dist-test-slave-39f2"
server_key: "d331789871791fad44958ce9ca7c61b3"
server_key_iv: "fe313b95005b336dc3731402aaa5cbf8"
server_key_version: "encryptionkey@0"
I20260927 05:00:24.262763 31689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:00:24.266369 31704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:24.267742 31689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:00:24.267926 31689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "d92580abfbc845b988c398cc343cf77f"
format_stamp: "Formatted at 2026-09-27 05:00:24 on dist-test-slave-39f2"
server_key: "d331789871791fad44958ce9ca7c61b3"
server_key_iv: "fe313b95005b336dc3731402aaa5cbf8"
server_key_version: "encryptionkey@0"
I20260927 05:00:24.268129 31689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:24.285055 31689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:24.289736 31689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:24.290088 31689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:24.291134 31689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:24.292625 31689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:24.292747 31689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:24.292933 31689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:24.293015 31689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:24.330895 31689 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:43723
I20260927 05:00:24.330927 31818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:43723 every 8 connection(s)
I20260927 05:00:24.332250 31689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:00:24.340417 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31689
I20260927 05:00:24.340624 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:00:24.341503 21157 external_mini_cluster.cc:1428] Setting key f91b52b25b5335876ebfa6c3e0564b99
Sep 27 05:00:24 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:00:24.355142 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:24.334896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:38245 (local address 127.20.169.126:34611)
0927 05:00:24.335377 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:24.335384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:24.336397 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 05:00:24.338147 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:24.338152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:24.338224 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:24.338377 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:24.339712 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.341014 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.342800 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.343390 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.343551 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:24.347421 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:24.347431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:24.347438 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:24.347497 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:24.351438 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:24.352274 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:24.352283 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:24.352290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:24.352464 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:24.353086 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:24.353091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:24.353093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:24.353422 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:24.353566 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:24.354657 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0927 05:00:24.354858 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260927 05:00:24.356951 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:24.335518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34611 (local address 127.20.169.65:38245)
0927 05:00:24.336221 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:24.336282 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:24.337864 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:24.338556 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:24.338563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:24.339035 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:24.339553 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:24.339569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.341170 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.341174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:24.342650 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:24.342664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.343623 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.343629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:24.343825 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:24.344542 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:24.344565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:24.347158 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:24.351603 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:00:24.351611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:00:24.351626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:00:24.352126 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:00:24.352602 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:00:24.352608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:00:24.352614 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:00:24.352933 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:00:24.353583 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:24.353595 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:24.354136 (+ 541us) client_negotiation.cc:769] Sending connection context
0927 05:00:24.354501 (+ 365us) client_negotiation.cc:240] Negotiation successful
0927 05:00:24.355165 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260927 05:00:24.358863 31819 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34611
I20260927 05:00:24.359202 31819 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:24.359997 31819 heartbeater.cc:511] Master 127.20.169.126:34611 requested a full tablet report, sending...
I20260927 05:00:24.362464 31627 ts_manager.cc:194] Registered new tserver with Master: d92580abfbc845b988c398cc343cf77f (127.20.169.65:43723)
I20260927 05:00:24.365003 31627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:38245
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:00:24.398391 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34611
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:00:24.729279 31825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:24.729637 31825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:24.729782 31825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:24.738137 31825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:24.738242 31825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:24.738353 31825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:00:24.747779 31825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:24.749183 31825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:24.750949 31825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:24.764752 31831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:24.764703 31833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:24.765381 31830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:24.766714 31825 server_base.cc:1034] running on GCE node
I20260927 05:00:24.767340 31825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:24.768394 31825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:24.769615 31825 hybrid_clock.cc:648] HybridClock initialized: now 1790485224769539 us; error 71 us; skew 500 ppm
I20260927 05:00:24.769942 31825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:24 dist-test-slave-39f2 krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:24.773252 31825 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:00:24.774781 31825 webserver.cc:460] Webserver started at http://127.20.169.66:45145/ using document root <none> and password file <none>
I20260927 05:00:24.775544 31825 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:24.775694 31825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:24.776008 31825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:24.778966 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "05204aefa4714a0194028be04ce60121"
format_stamp: "Formatted at 2026-09-27 05:00:24 on dist-test-slave-39f2"
server_key: "6ea0892a7109dddcb79a63167820f561"
server_key_iv: "23c9a5daa13385fe18fbf99c200ed1a6"
server_key_version: "encryptionkey@0"
I20260927 05:00:24.779723 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "05204aefa4714a0194028be04ce60121"
format_stamp: "Formatted at 2026-09-27 05:00:24 on dist-test-slave-39f2"
server_key: "6ea0892a7109dddcb79a63167820f561"
server_key_iv: "23c9a5daa13385fe18fbf99c200ed1a6"
server_key_version: "encryptionkey@0"
I20260927 05:00:24.784776 31825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:00:24.788116 31840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:24.789538 31825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:00:24.789707 31825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "05204aefa4714a0194028be04ce60121"
format_stamp: "Formatted at 2026-09-27 05:00:24 on dist-test-slave-39f2"
server_key: "6ea0892a7109dddcb79a63167820f561"
server_key_iv: "23c9a5daa13385fe18fbf99c200ed1a6"
server_key_version: "encryptionkey@0"
I20260927 05:00:24.789945 31825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:24.824887 31825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:24.829349 31825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:24.829654 31825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:24.830658 31825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:24.832022 31825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:24.832124 31825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:24.832278 31825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:24.832348 31825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:24.865795 31825 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:40565
I20260927 05:00:24.865882 31952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:40565 every 8 connection(s)
I20260927 05:00:24.867179 31825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:00:24.871766 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31825
I20260927 05:00:24.871940 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:00:24.872493 21157 external_mini_cluster.cc:1428] Setting key 448aa3005b23f7f69db0493c520adf4b
Sep 27 05:00:24 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:00:24.890906 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:24.870076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:51713 (local address 127.20.169.126:34611)
0927 05:00:24.870605 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:24.870609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:24.871494 (+ 885us) server_negotiation.cc:407] Connection header received
0927 05:00:24.873809 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:24.873815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:24.873905 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:24.874023 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:24.876131 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.876985 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.879451 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.879890 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.879993 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:24.884533 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:24.884539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:24.884542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:24.884575 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:24.887436 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:24.888409 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:24.888413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:24.888416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:24.888552 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:24.889102 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:24.889106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:24.889108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:24.889391 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:24.889551 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:24.890466 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 05:00:24.890636 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":313,"threads_started":1}
I20260927 05:00:24.892423 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:24.870595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34611 (local address 127.20.169.66:51713)
0927 05:00:24.871312 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:24.871362 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:24.873536 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:24.874263 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:24.874271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:24.875067 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:24.875930 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:24.875959 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.877144 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.877153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:24.879293 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:24.879312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:24.880128 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:24.880138 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:24.880480 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:24.881267 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:24.881305 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:24.884275 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:24.887586 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:00:24.887597 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:00:24.887616 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:00:24.888258 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:00:24.888664 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:00:24.888671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:00:24.888677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:00:24.888969 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:00:24.889564 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:24.889575 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:24.890029 (+ 454us) client_negotiation.cc:769] Sending connection context
0927 05:00:24.890345 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 05:00:24.890931 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":272,"threads_started":1}
I20260927 05:00:24.894193 31953 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34611
I20260927 05:00:24.894536 31953 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:24.895254 31953 heartbeater.cc:511] Master 127.20.169.126:34611 requested a full tablet report, sending...
I20260927 05:00:24.896906 31627 ts_manager.cc:194] Registered new tserver with Master: 05204aefa4714a0194028be04ce60121 (127.20.169.66:40565)
I20260927 05:00:24.897701 31627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:51713
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:00:24.930426 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34611
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:00:25.255824 31959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:00:25.256165 31959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:00:25.256330 31959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:00:25.263660 31959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:00:25.263808 31959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:00:25.264008 31959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:00:25.273960 31959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:00:25.275314 31959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:00:25.276626 31959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:00:25.292987 31965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:25.293027 31964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:00:25.293578 31968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:00:25.293934 31959 server_base.cc:1034] running on GCE node
I20260927 05:00:25.294670 31959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:00:25.295588 31959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:00:25.296782 31959 hybrid_clock.cc:648] HybridClock initialized: now 1790485225296739 us; error 40 us; skew 500 ppm
I20260927 05:00:25.297200 31959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:00:25 dist-test-slave-39f2 krb5kdc[31581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485225, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:00:25.300172 31959 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:00:25.301648 31959 webserver.cc:460] Webserver started at http://127.20.169.67:34291/ using document root <none> and password file <none>
I20260927 05:00:25.302325 31959 fs_manager.cc:359] Metadata directory not provided
I20260927 05:00:25.302450 31959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:00:25.302784 31959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:00:25.305604 31959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "b538623f9705496cb560feab3fcdcdcb"
format_stamp: "Formatted at 2026-09-27 05:00:25 on dist-test-slave-39f2"
server_key: "04fd46df760bb8f1ce812aff9ae1021a"
server_key_iv: "10e919261e566c0ba0489b6ec43d534c"
server_key_version: "encryptionkey@0"
I20260927 05:00:25.306344 31959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "b538623f9705496cb560feab3fcdcdcb"
format_stamp: "Formatted at 2026-09-27 05:00:25 on dist-test-slave-39f2"
server_key: "04fd46df760bb8f1ce812aff9ae1021a"
server_key_iv: "10e919261e566c0ba0489b6ec43d534c"
server_key_version: "encryptionkey@0"
I20260927 05:00:25.311306 31959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:00:25.315047 31975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:25.316362 31959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:00:25.316519 31959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "b538623f9705496cb560feab3fcdcdcb"
format_stamp: "Formatted at 2026-09-27 05:00:25 on dist-test-slave-39f2"
server_key: "04fd46df760bb8f1ce812aff9ae1021a"
server_key_iv: "10e919261e566c0ba0489b6ec43d534c"
server_key_version: "encryptionkey@0"
I20260927 05:00:25.316687 31959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:00:25.333575 31959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:00:25.337572 31959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:00:25.337813 31959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:00:25.338606 31959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:00:25.339816 31959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:00:25.339886 31959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:25.339990 31959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:00:25.340034 31959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:00:25.368360 31819 heartbeater.cc:503] Master 127.20.169.126:34611 was elected leader, sending a full tablet report...
I20260927 05:00:25.373106 31959 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:46695
I20260927 05:00:25.373195 32087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:46695 every 8 connection(s)
I20260927 05:00:25.374455 31959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:00:25.383781 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 31959
I20260927 05:00:25.383999 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:00:25.384490 21157 external_mini_cluster.cc:1428] Setting key 2ed76cf55c2192dbe4ab00d5b0cb2830
Sep 27 05:00:25 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485225, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:00:25.394179 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.377077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:47857 (local address 127.20.169.126:34611)
0927 05:00:25.377258 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.377262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.378529 (+ 1267us) server_negotiation.cc:407] Connection header received
0927 05:00:25.380223 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.380228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.380298 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.380400 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:25.381908 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.382614 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.384149 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.384528 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.384657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.388419 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:25.388424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:25.388428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:25.388459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:25.391202 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:25.391926 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:25.391929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:25.391931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:25.392057 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:25.392515 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:25.392517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:25.392519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:25.392760 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:25.392950 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.393778 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.393942 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:00:25.395236 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.377655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34611 (local address 127.20.169.67:47857)
0927 05:00:25.378361 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:25.378403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:25.379962 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:25.380608 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:25.380615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:25.381182 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:00:25.381743 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.381760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.382752 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.382755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.383993 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.384016 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.384760 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.384767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.385011 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.385760 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:00:25.385785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:00:25.388247 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:00:25.391337 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:00:25.391345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:00:25.391357 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:00:25.391794 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:00:25.392180 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:00:25.392184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:00:25.392187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:00:25.392402 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:00:25.392929 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:00:25.392935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:00:25.393356 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 05:00:25.393669 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 05:00:25.394079 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":293,"threads_started":1}
I20260927 05:00:25.396590 32088 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34611
I20260927 05:00:25.396934 32088 heartbeater.cc:464] Registering TS with master...
I20260927 05:00:25.397670 32088 heartbeater.cc:511] Master 127.20.169.126:34611 requested a full tablet report, sending...
I20260927 05:00:25.399197 31627 ts_manager.cc:194] Registered new tserver with Master: b538623f9705496cb560feab3fcdcdcb (127.20.169.67:46695)
I20260927 05:00:25.400223 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:00:25.400250 31627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:47857
I20260927 05:00:25.416460 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.404750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.20.169.126:34611)
0927 05:00:25.405011 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.405020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.405347 (+ 327us) server_negotiation.cc:407] Connection header received
0927 05:00:25.405472 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.405475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.405550 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.405625 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:25.406963 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.407641 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.408926 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.409316 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.409406 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.411467 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:25.411471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:25.411474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:25.411509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:25.414208 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:25.414796 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:25.414799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:25.414801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:25.414918 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:25.415228 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:25.415231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:25.415233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:25.415517 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:25.415697 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.415966 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.416108 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 05:00:25.420300 31627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:00:25.426972 31627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:00:25.449411 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.438818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:40565 (local address 127.0.0.1:34546)
0927 05:00:25.440335 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:25.440397 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:25.440604 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:25.441710 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:25.441722 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:25.441766 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:00:25.442337 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.442352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.444481 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.444486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.446292 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.446306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.448213 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.448220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.448564 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.448638 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 05:00:25.448779 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 05:00:25.449008 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"mutex_wait_us":68,"thread_start_us":1024,"threads_started":1}
I20260927 05:00:25.449373 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.438900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.20.169.66:40565)
0927 05:00:25.440997 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.441009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.441042 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:00:25.441154 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.441165 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.441448 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.442563 (+ 1115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:00:25.442777 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.444347 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.446503 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.447993 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.448138 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.448343 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.448848 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.449035 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1864,"thread_start_us":1730,"threads_started":1}
I20260927 05:00:25.452276 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.441410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43723 (local address 127.0.0.1:40344)
0927 05:00:25.443245 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:25.443269 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:25.443451 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:25.444233 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:25.444239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:25.444269 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:00:25.444700 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.444712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.446497 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.446502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.448520 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.448537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.451182 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.451189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.451484 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.451668 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 05:00:25.451745 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 05:00:25.451887 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"mutex_wait_us":215,"thread_start_us":1558,"threads_started":1}
I20260927 05:00:25.452320 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.439494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.20.169.67:46695)
0927 05:00:25.441966 (+ 2472us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.442001 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.442757 (+ 756us) server_negotiation.cc:407] Connection header received
0927 05:00:25.442885 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.442897 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.443144 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.443332 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:00:25.444182 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.445904 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.448772 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.450640 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.450767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.450996 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.451856 (+ 860us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.452016 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2020,"thread_start_us":1875,"threads_started":1}
I20260927 05:00:25.452316 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.442008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.20.169.65:43723)
0927 05:00:25.442797 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.442816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.443355 (+ 539us) server_negotiation.cc:407] Connection header received
0927 05:00:25.443612 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.443619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.443846 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.443990 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:00:25.444936 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.446304 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.448690 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.450640 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.450767 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.450996 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.451817 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.451993 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":481,"threads_started":1}
I20260927 05:00:25.452934 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.438934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:46695 (local address 127.0.0.1:37910)
0927 05:00:25.442490 (+ 3556us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:25.442517 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:25.442629 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:25.443592 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:25.443597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:25.443622 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:00:25.444000 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.444010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.446049 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.446054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.448566 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.448578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.451196 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.451201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.451481 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.451578 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 05:00:25.451668 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 05:00:25.452554 (+ 886us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3403,"mutex_wait_us":965,"thread_start_us":3232,"threads_started":1}
I20260927 05:00:25.453615 31888 tablet_service.cc:1467] Processing CreateTablet for tablet 99fd19b90bc044de90958e0f515fba87 (DEFAULT_TABLE table=test-table [id=6590b10c9fa045c6852e664791da365f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:00:25.455138 31888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fd19b90bc044de90958e0f515fba87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:00:25.456315 31753 tablet_service.cc:1467] Processing CreateTablet for tablet 99fd19b90bc044de90958e0f515fba87 (DEFAULT_TABLE table=test-table [id=6590b10c9fa045c6852e664791da365f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:00:25.456640 32023 tablet_service.cc:1467] Processing CreateTablet for tablet 99fd19b90bc044de90958e0f515fba87 (DEFAULT_TABLE table=test-table [id=6590b10c9fa045c6852e664791da365f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:00:25.457751 31753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fd19b90bc044de90958e0f515fba87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:00:25.457989 32023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99fd19b90bc044de90958e0f515fba87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:00:25.469712 32106 tablet_bootstrap.cc:492] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Bootstrap starting.
I20260927 05:00:25.470697 32107 tablet_bootstrap.cc:492] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Bootstrap starting.
I20260927 05:00:25.473711 32106 tablet_bootstrap.cc:654] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Neither blocks nor log segments found. Creating new log.
I20260927 05:00:25.474251 32107 tablet_bootstrap.cc:654] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Neither blocks nor log segments found. Creating new log.
I20260927 05:00:25.475122 32106 log.cc:824] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:25.475483 32107 log.cc:824] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:25.479385 32106 tablet_bootstrap.cc:492] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: No bootstrap required, opened a new log
I20260927 05:00:25.479738 32106 ts_tablet_manager.cc:1397] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260927 05:00:25.481053 32108 tablet_bootstrap.cc:492] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Bootstrap starting.
I20260927 05:00:25.483767 32108 tablet_bootstrap.cc:654] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Neither blocks nor log segments found. Creating new log.
I20260927 05:00:25.485098 32108 log.cc:824] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:00:25.487293 32107 tablet_bootstrap.cc:492] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: No bootstrap required, opened a new log
I20260927 05:00:25.487589 32107 ts_tablet_manager.cc:1397] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260927 05:00:25.487638 32108 tablet_bootstrap.cc:492] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: No bootstrap required, opened a new log
I20260927 05:00:25.488168 32108 ts_tablet_manager.cc:1397] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260927 05:00:25.491580 32106 raft_consensus.cc:348] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.492034 32106 raft_consensus.cc:374] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:00:25.492130 32106 raft_consensus.cc:729] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b538623f9705496cb560feab3fcdcdcb, State: Initialized, Role: FOLLOWER
I20260927 05:00:25.492740 32106 consensus_queue.cc:260] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.493906 32088 heartbeater.cc:503] Master 127.20.169.126:34611 was elected leader, sending a full tablet report...
I20260927 05:00:25.495335 32106 ts_tablet_manager.cc:1428] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 05:00:25.496521 32108 raft_consensus.cc:348] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.496892 32108 raft_consensus.cc:374] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:00:25.497013 32108 raft_consensus.cc:729] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92580abfbc845b988c398cc343cf77f, State: Initialized, Role: FOLLOWER
I20260927 05:00:25.497789 32108 consensus_queue.cc:260] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.497926 32107 raft_consensus.cc:348] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.498360 32107 raft_consensus.cc:374] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:00:25.498479 32107 raft_consensus.cc:729] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05204aefa4714a0194028be04ce60121, State: Initialized, Role: FOLLOWER
I20260927 05:00:25.499744 32108 ts_tablet_manager.cc:1428] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260927 05:00:25.500341 32107 consensus_queue.cc:260] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.501746 31953 heartbeater.cc:503] Master 127.20.169.126:34611 was elected leader, sending a full tablet report...
I20260927 05:00:25.502308 32107 ts_tablet_manager.cc:1428] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260927 05:00:25.584904 31820 tablet.cc:2369] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:00:25.619431 31954 tablet.cc:2369] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:00:25.626179 32089 tablet.cc:2369] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:00:25.731027 32114 raft_consensus.cc:482] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:00:25.731385 32114 raft_consensus.cc:504] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.733511 32114 leader_election.cc:288] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b538623f9705496cb560feab3fcdcdcb (127.20.169.67:46695), d92580abfbc845b988c398cc343cf77f (127.20.169.65:43723)
I20260927 05:00:25.740800 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.734110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:52433 (local address 127.20.169.65:43723)
0927 05:00:25.734360 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.734369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.735229 (+ 860us) server_negotiation.cc:407] Connection header received
0927 05:00:25.735476 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.735482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.735563 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.735667 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:00:25.736352 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.737166 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.738962 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.739745 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.739930 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.740018 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.740406 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.740537 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:00:25.740782 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.734022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:41937 (local address 127.20.169.67:46695)
0927 05:00:25.734253 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.734262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.735101 (+ 839us) server_negotiation.cc:407] Connection header received
0927 05:00:25.735264 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.735270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.735349 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.735442 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:00:25.736099 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.736980 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.738994 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.739760 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.739954 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.740030 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.740408 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.740536 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:00:25.740940 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.733996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:43723 (local address 127.20.169.66:52433)
0927 05:00:25.735095 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:25.735115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:25.735264 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:25.735690 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:25.735693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:25.735711 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:00:25.736178 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.736191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.737317 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.737320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.738781 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.738794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.739987 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.739997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.740194 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.740235 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 05:00:25.740359 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 05:00:25.740485 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":55,"thread_start_us":698,"threads_started":1}
I20260927 05:00:25.741457 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.733969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:46695 (local address 127.20.169.66:41937)
0927 05:00:25.734968 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 05:00:25.734988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:00:25.735124 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:00:25.735471 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:00:25.735475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:00:25.735501 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:00:25.735937 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.735948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.737097 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.737102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.738807 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:00:25.738819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.740014 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.740020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:00:25.740196 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.740228 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:00:25.740374 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 05:00:25.740500 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":24,"thread_start_us":523,"threads_started":1}
I20260927 05:00:25.742056 31773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fd19b90bc044de90958e0f515fba87" candidate_uuid: "05204aefa4714a0194028be04ce60121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92580abfbc845b988c398cc343cf77f" is_pre_election: true
I20260927 05:00:25.742527 32043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fd19b90bc044de90958e0f515fba87" candidate_uuid: "05204aefa4714a0194028be04ce60121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b538623f9705496cb560feab3fcdcdcb" is_pre_election: true
I20260927 05:00:25.742650 31773 raft_consensus.cc:2454] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05204aefa4714a0194028be04ce60121 in term 0.
I20260927 05:00:25.743057 32043 raft_consensus.cc:2454] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05204aefa4714a0194028be04ce60121 in term 0.
I20260927 05:00:25.743367 31843 leader_election.cc:302] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05204aefa4714a0194028be04ce60121, d92580abfbc845b988c398cc343cf77f; no voters:
I20260927 05:00:25.743799 32114 raft_consensus.cc:2781] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:00:25.743976 32114 raft_consensus.cc:482] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:00:25.744087 32114 raft_consensus.cc:3037] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:00:25.746326 32114 raft_consensus.cc:504] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.746886 32114 leader_election.cc:288] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [CANDIDATE]: Term 1 election: Requested vote from peers b538623f9705496cb560feab3fcdcdcb (127.20.169.67:46695), d92580abfbc845b988c398cc343cf77f (127.20.169.65:43723)
I20260927 05:00:25.747366 32043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fd19b90bc044de90958e0f515fba87" candidate_uuid: "05204aefa4714a0194028be04ce60121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b538623f9705496cb560feab3fcdcdcb"
I20260927 05:00:25.747416 31773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99fd19b90bc044de90958e0f515fba87" candidate_uuid: "05204aefa4714a0194028be04ce60121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92580abfbc845b988c398cc343cf77f"
I20260927 05:00:25.747576 32043 raft_consensus.cc:3037] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:00:25.747597 31773 raft_consensus.cc:3037] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:00:25.750737 32043 raft_consensus.cc:2454] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05204aefa4714a0194028be04ce60121 in term 1.
I20260927 05:00:25.750737 31773 raft_consensus.cc:2454] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05204aefa4714a0194028be04ce60121 in term 1.
I20260927 05:00:25.751171 31842 leader_election.cc:302] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05204aefa4714a0194028be04ce60121, b538623f9705496cb560feab3fcdcdcb; no voters:
I20260927 05:00:25.751495 32114 raft_consensus.cc:2781] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:00:25.751950 32114 raft_consensus.cc:686] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 1 LEADER]: Becoming Leader. State: Replica: 05204aefa4714a0194028be04ce60121, State: Running, Role: LEADER
I20260927 05:00:25.752526 32114 consensus_queue.cc:237] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } }
I20260927 05:00:25.757826 31626 catalog_manager.cc:5495] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05204aefa4714a0194028be04ce60121 (127.20.169.66). New cstate: current_term: 1 leader_uuid: "05204aefa4714a0194028be04ce60121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05204aefa4714a0194028be04ce60121" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 } health_report { overall_health: UNKNOWN } } }
I20260927 05:00:25.816033 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.810668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.20.169.66:40565)
0927 05:00:25.810886 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.810891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.810911 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:00:25.811073 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.811077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.811168 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.811309 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:00:25.811654 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.812426 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.813853 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.814272 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.814394 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.814874 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:00:25.815481 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:00:25.815599 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.815712 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.815799 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:00:25.824359 31773 raft_consensus.cc:1261] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 1 FOLLOWER]: Refusing update from remote peer 05204aefa4714a0194028be04ce60121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:00:25.824365 32043 raft_consensus.cc:1261] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 1 FOLLOWER]: Refusing update from remote peer 05204aefa4714a0194028be04ce60121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:00:25.825304 32118 consensus_queue.cc:1035] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92580abfbc845b988c398cc343cf77f" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:00:25.825661 32114 consensus_queue.cc:1035] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b538623f9705496cb560feab3fcdcdcb" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:00:25.836516 32121 mvcc.cc:205] Tried to move back new op lower bound from 7333827484967055360 to 7333827484686991360. Current Snapshot: MvccSnapshot[applied={T|T < 7333827484967055360}]
I20260927 05:00:25.839176 32122 mvcc.cc:205] Tried to move back new op lower bound from 7333827484967055360 to 7333827484686991360. Current Snapshot: MvccSnapshot[applied={T|T < 7333827484967055360}]
I20260927 05:00:25.853049 31626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:00:25.853365 31626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:00:25.857239 31626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 447403f7849a463aa60fdbe5fab99f65: Sending DeleteTablet for 3 replicas of tablet 99fd19b90bc044de90958e0f515fba87
I20260927 05:00:25.858356 31888 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fd19b90bc044de90958e0f515fba87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:00:25 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34546
I20260927 05:00:25.858465 32023 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fd19b90bc044de90958e0f515fba87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:00:25 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:37910
I20260927 05:00:25.858911 31753 tablet_service.cc:1514] Processing DeleteTablet for tablet 99fd19b90bc044de90958e0f515fba87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:00:25 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:40344
I20260927 05:00:25.860922 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31689
I20260927 05:00:25.890470 32132 tablet_replica.cc:330] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: stopping tablet replica
I20260927 05:00:25.891275 32132 raft_consensus.cc:2229] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:00:25.892071 32132 raft_consensus.cc:2258] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:00:25.893417 32133 tablet_replica.cc:330] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: stopping tablet replica
I20260927 05:00:25.893882 32133 raft_consensus.cc:2229] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:00:25.894445 32133 raft_consensus.cc:2258] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:00:25.895069 32132 ts_tablet_manager.cc:1905] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:00:25.895315 32131 tablet_replica.cc:330] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: stopping tablet replica
I20260927 05:00:25.895699 32131 raft_consensus.cc:2229] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:00:25.900295 32133 ts_tablet_manager.cc:1905] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:00:25.901003 32131 raft_consensus.cc:2258] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:00:25.904060 32132 ts_tablet_manager.cc:1918] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:00:25.904241 32132 log.cc:1196] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/99fd19b90bc044de90958e0f515fba87
I20260927 05:00:25.904783 32132 ts_tablet_manager.cc:1939] T 99fd19b90bc044de90958e0f515fba87 P b538623f9705496cb560feab3fcdcdcb: Deleting consensus metadata
Sep 27 05:00:25 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:00:25.906639 31611 catalog_manager.cc:4841] TS b538623f9705496cb560feab3fcdcdcb (127.20.169.67:46695): tablet 99fd19b90bc044de90958e0f515fba87 (table test-table [id=6590b10c9fa045c6852e664791da365f]) successfully deleted
I20260927 05:00:25.907686 32133 ts_tablet_manager.cc:1918] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:00:25.907868 32133 log.cc:1196] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/99fd19b90bc044de90958e0f515fba87
I20260927 05:00:25.908386 32133 ts_tablet_manager.cc:1939] T 99fd19b90bc044de90958e0f515fba87 P d92580abfbc845b988c398cc343cf77f: Deleting consensus metadata
I20260927 05:00:25.909863 32131 ts_tablet_manager.cc:1905] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:00:25.910223 31612 catalog_manager.cc:4841] TS d92580abfbc845b988c398cc343cf77f (127.20.169.65:43723): tablet 99fd19b90bc044de90958e0f515fba87 (table test-table [id=6590b10c9fa045c6852e664791da365f]) successfully deleted
I20260927 05:00:25.913370 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:25.862218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.20.169.65:43723)
0927 05:00:25.862433 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:25.862439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:25.884363 (+ 21924us) server_negotiation.cc:407] Connection header received
0927 05:00:25.889057 (+ 4694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:25.889065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:25.889164 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:25.889244 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:25.892709 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.894016 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.896278 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:25.897030 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:25.897127 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:25.905975 (+ 8848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:25.905990 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:25.906014 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:25.906057 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:25.910092 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:25.910860 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:25.910868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:25.910872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:25.911015 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:25.911457 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:25.911462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:25.911465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:25.911926 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:25.912194 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:25.912748 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 05:00:25.912987 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:00:25.914786 31814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:25.918174 32131 ts_tablet_manager.cc:1918] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:00:25.918355 32131 log.cc:1196] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/99fd19b90bc044de90958e0f515fba87
I20260927 05:00:25.919030 32131 ts_tablet_manager.cc:1939] T 99fd19b90bc044de90958e0f515fba87 P 05204aefa4714a0194028be04ce60121: Deleting consensus metadata
I20260927 05:00:25.920984 31610 catalog_manager.cc:4841] TS 05204aefa4714a0194028be04ce60121 (127.20.169.66:40565): tablet 99fd19b90bc044de90958e0f515fba87 (table test-table [id=6590b10c9fa045c6852e664791da365f]) successfully deleted
I20260927 05:00:26.009290 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31689
I20260927 05:00:26.028612 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31825
Sep 27 05:00:26 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:00:26.047197 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:26.029516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.20.169.66:40565)
0927 05:00:26.029679 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:26.029683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:26.029706 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:00:26.029912 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:26.029916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:26.029982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:26.030079 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:26.031682 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:26.032529 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:26.034382 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:26.034826 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:26.034927 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:26.040397 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:26.040416 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:26.040432 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:26.040480 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:26.044053 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:26.044949 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:26.044957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:26.044960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:26.045111 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:26.045563 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:26.045567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:26.045569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:26.045942 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:26.046232 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:26.046571 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 05:00:26.046752 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:00:26.048100 31948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:26.165148 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31825
I20260927 05:00:26.186184 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31959
Sep 27 05:00:26 dist-test-slave-39f2 krb5kdc[31581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:00:26.201071 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:00:26.186975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.20.169.67:46695)
0927 05:00:26.187176 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:00:26.187180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:00:26.187221 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:00:26.187418 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:00:26.187423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:00:26.187492 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:00:26.187584 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:00:26.188924 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:26.189634 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:26.190868 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:00:26.191239 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:00:26.191345 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:00:26.195487 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:00:26.195500 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:00:26.195516 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:00:26.195553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:00:26.198406 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:26.199039 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:26.199052 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:26.199054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:26.199194 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:00:26.199569 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:00:26.199572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:00:26.199574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:00:26.199971 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:00:26.200217 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 05:00:26.200509 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 05:00:26.200666 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 05:00:26.201767 32083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:26.290115 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31959
I20260927 05:00:26.307716 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 31594
I20260927 05:00:26.309015 31658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:00:26.418007 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 31594
2026-09-27T05:00:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4495 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:00:26 dist-test-slave-39f2 krb5kdc[32141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:00:26 dist-test-slave-39f2 krb5kdc[32141](info): set up 2 sockets
Sep 27 05:00:26 dist-test-slave-39f2 krb5kdc[32141](info): commencing operation
krb5kdc: starting...
W20260927 05:00:27.541970 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:00:27 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:00:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/rangeradmin_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/rangeradmin_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/rangerlookup_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/rangerlookup_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
I20260927 05:00:27.886148 21157 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRTYp1d/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/postgres -l logfile start
2026-09-27 05:00:30.466 UTC [32171] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 05:00:30.466 UTC [32171] LOG: listening on IPv4 address "127.20.169.84", port 42409
2026-09-27 05:00:30.469 UTC [32171] LOG: listening on Unix socket "/tmp/.s.PGSQL.42409"
2026-09-27 05:00:30.490 UTC [32174] LOG: database system was shut down at 2026-09-27 05:00:29 UTC
2026-09-27 05:00:30.494 UTC [32171] LOG: database system is ready to accept connections
I20260927 05:00:31.481348 21157 mini_postgres.cc:96] Postgres bound to 42409
2026-09-27 05:00:31.501 UTC [32182] FATAL: database "slave" does not exist
127.20.169.84:42409 - accepting connections
I20260927 05:00:31.502594 21157 mini_ranger.cc:161] Starting Ranger...
I20260927 05:00:31.531662 21157 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 05:00:32.008129 21157 mini_ranger.cc:87] Created ranger Postgres database
I20260927 05:00:32.008260 21157 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-admin
2026-09-27 05:00:32,382 [I] DB FLAVOR :POSTGRES
2026-09-27 05:00:32,383 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:00:32,383 [I] Checking connection..
2026-09-27 05:00:32,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 05:00:32,682 [I] Checking connection passed.
2026-09-27 05:00:32,683 [I] --------- Verifying version history table ---------
2026-09-27 05:00:32,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:00:32,930 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:00:32,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:00:33,183 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:00:33,184 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 05:00:33,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 05:00:33.438 UTC [32246] WARNING: there is no transaction in progress
2026-09-27 05:00:33,444 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 05:00:33,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:00:33,694 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 05:00:33,695 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:00:33,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 05:00:33,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 05:00:34,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-39f2.c.gcp-upstream.internal','N') ;"
2026-09-27 05:00:34,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:00:34,686 [I] Table x_portal_user does not exist in database ranger
2026-09-27 05:00:34,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:00:34,926 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 05:00:34,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:00:35,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:00:35,435 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 05:00:35,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 05:00:36.384 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.400 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.416 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.431 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.446 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.464 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.520 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.527 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.542 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.552 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.563 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.578 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.588 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.594 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.601 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:36.607 UTC [32355] WARNING: there is no transaction in progress
2026-09-27 05:00:37,248 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 05:00:37,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 05:00:37,507 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 05:00:37,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:00:37,764 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 05:00:37,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:00:38,013 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 05:00:38,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:00:38,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:00:38,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-39f2.c.gcp-upstream.internal';"
2026-09-27 05:00:38,764 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 05:00:38,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 05:00:39,018 [I] DB_PATCHES have already been applied
I20260927 05:00:39.024915 21157 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRTYp1d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 05:00:39.025027 21157 mini_ranger.cc:193] Using host: 127.20.169.84
I20260927 05:00:39.041551 21157 mini_ranger.cc:227] Ranger admin URL: http://127.20.169.84:39185
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 5:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 05:00:40.178804 21157 mini_ranger.cc:160] Time spent starting Ranger: real 8.676s user 0.008s sys 0.036s
Sep 27, 2026 5:00:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 5:00:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 5:00:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 5:00:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 5:01:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 05:01:01.331872 21157 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/rangerkms_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/rangerkms_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.169.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.169.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 05:01:01 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 05:01:01.513639 21157 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 05:01:01.537045 21157 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 05:01:02.406387 21157 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 05:01:02.406523 21157 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms
I20260927 05:01:02.406666 21157 mini_ranger_kms.cc:227] Using postgres at 127.20.169.84:42409
2026-09-27 05:01:02,910 [I] DB FLAVOR :POSTGRES
2026-09-27 05:01:02,911 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:01:02,911 [I] Checking connection
2026-09-27 05:01:02,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 05:01:03,170 [I] connection success
2026-09-27 05:01:03,170 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 05:01:03,171 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 05:01:03,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 05:01:03,444 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 05:01:03,445 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:01:03,445 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 05:01:03,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:42409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 05:01:03,698 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 05:01:04.146880 21157 mini_ranger_kms.cc:310] Created kms service
I20260927 05:01:04.323575 21157 mini_ranger_kms.cc:326] Created kudu user
I20260927 05:01:04.370939 21157 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 05:01:04 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:04.782387 21157 mini_ranger_kms.cc:384] Added ranger policy
I20260927 05:01:04.782734 21157 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 05:01:04.782814 21157 mini_ranger_kms.cc:240] Using host: 127.20.169.84
I20260927 05:01:04.805313 21157 mini_ranger_kms.cc:276] Ranger KMS PID: 32537
I20260927 05:01:04.805511 21157 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.169.84:37659
Sep 27, 2026 5:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 5:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 5:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 5:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 05:01:07.096520 21157 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.583s user 0.123s sys 0.408s
I20260927 05:01:07.096796 21157 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 05:01:07.097078 21157 mini_ranger_kms.cc:401] 127.20.169.84:37659/kms/v1/keys
Sep 27 05:01:07 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.169.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 05:01:07 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.169.84@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
Sep 27 05:01:07 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.169.84@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
Sep 27 05:01:10 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:01:10.356998 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:43633
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:43633
--ranger_config_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:37659/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:01:10.858654 32588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:01:10.858999 32588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:01:10.859169 32588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:01:10.866854 32588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:01:10.867005 32588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:01:10.867100 32588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:01:10.867203 32588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:01:10.867290 32588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:01:10.877552 32588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.169.84:37659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:43633
--ranger_config_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:01:10.878994 32588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:01:10.880359 32588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:01:10.893532 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:10.893499 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:10.894217 32593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:01:10.898474 32588 server_base.cc:1034] running on GCE node
I20260927 05:01:10.899291 32588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:01:10.900851 32588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:01:10.902081 32588 hybrid_clock.cc:648] HybridClock initialized: now 1790485270902028 us; error 60 us; skew 500 ppm
I20260927 05:01:10.902444 32588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:01:10 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:01:10.905586 32588 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:01:10.907186 32588 webserver.cc:460] Webserver started at http://127.20.169.126:35775/ using document root <none> and password file <none>
I20260927 05:01:10.907994 32588 fs_manager.cc:359] Metadata directory not provided
I20260927 05:01:10.908123 32588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:01:10.908461 32588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:01:10 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:10.974819 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "c78be62e69de4142b2d92150ce030a22"
format_stamp: "Formatted at 2026-09-27 05:01:10 on dist-test-slave-39f2"
server_key: "d0623275ac1e7d43b05c9dd3bdcbc8b5"
server_key_iv: "efa6f5197d78b180e18fed2d545764d2"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:10.975699 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "c78be62e69de4142b2d92150ce030a22"
format_stamp: "Formatted at 2026-09-27 05:01:10 on dist-test-slave-39f2"
server_key: "d0623275ac1e7d43b05c9dd3bdcbc8b5"
server_key_iv: "efa6f5197d78b180e18fed2d545764d2"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:10.981515 32588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 05:01:11.002509 32603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:11.004221 32588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 05:01:11.004439 32588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "c78be62e69de4142b2d92150ce030a22"
format_stamp: "Formatted at 2026-09-27 05:01:10 on dist-test-slave-39f2"
server_key: "d0623275ac1e7d43b05c9dd3bdcbc8b5"
server_key_iv: "efa6f5197d78b180e18fed2d545764d2"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:11.004683 32588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:01:11.032730 32588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:01:11.037647 32588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:01:11.038125 32588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:01:11.071363 32588 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:43633
I20260927 05:01:11.071382 32655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:43633 every 8 connection(s)
I20260927 05:01:11.072940 32588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:01:11.078562 32656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:01:11.082445 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 32588
I20260927 05:01:11.082670 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:01:11.083338 21157 external_mini_cluster.cc:1428] Setting key fa48185f863457699a76b7f997e1e29f
I20260927 05:01:11.103044 32656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22: Bootstrap starting.
Sep 27 05:01:11 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:01:11.108139 32656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22: Neither blocks nor log segments found. Creating new log.
I20260927 05:01:11.110955 32656 log.cc:824] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22: Log is configured to *not* fsync() on all Append() calls
I20260927 05:01:11.116379 32656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22: No bootstrap required, opened a new log
I20260927 05:01:11.121882 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:11.085807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.20.169.126:43633)
0927 05:01:11.090023 (+ 4216us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:11.090050 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:11.090152 (+ 102us) server_negotiation.cc:407] Connection header received
0927 05:01:11.092934 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:11.092990 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:11.093704 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:11.094342 (+ 638us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:11.096085 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:11.098995 (+ 2910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:11.100998 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:11.102442 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:11.102652 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:11.108221 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:11.108238 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:11.108261 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:11.108359 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:11.114308 (+ 5949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:11.115417 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:11.115425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:11.115434 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:11.115682 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:11.116217 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:11.116222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:11.116225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:11.117343 (+ 1118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:11.118918 (+ 1575us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:11.119254 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 05:01:11.119861 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3800,"thread_start_us":3319,"threads_started":1}
I20260927 05:01:11.130559 32656 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be62e69de4142b2d92150ce030a22" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43633 } }
I20260927 05:01:11.131217 32656 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:01:11.131371 32656 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78be62e69de4142b2d92150ce030a22, State: Initialized, Role: FOLLOWER
I20260927 05:01:11.132172 32656 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be62e69de4142b2d92150ce030a22" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43633 } }
I20260927 05:01:11.132619 32656 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:01:11.132882 32656 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:01:11.133162 32656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:01:11.136178 32656 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be62e69de4142b2d92150ce030a22" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43633 } }
I20260927 05:01:11.136864 32656 leader_election.cc:302] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78be62e69de4142b2d92150ce030a22; no voters:
I20260927 05:01:11.137881 32656 leader_election.cc:288] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:01:11.138195 32661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:01:11.141392 32656 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:01:11.141875 32661 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [term 1 LEADER]: Becoming Leader. State: Replica: c78be62e69de4142b2d92150ce030a22, State: Running, Role: LEADER
I20260927 05:01:11.142784 32661 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be62e69de4142b2d92150ce030a22" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43633 } }
I20260927 05:01:11.150309 32663 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78be62e69de4142b2d92150ce030a22. Latest consensus state: current_term: 1 leader_uuid: "c78be62e69de4142b2d92150ce030a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be62e69de4142b2d92150ce030a22" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43633 } } }
I20260927 05:01:11.151099 32663 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [sys.catalog]: This master's current role is: LEADER
I20260927 05:01:11.153426 32662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78be62e69de4142b2d92150ce030a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78be62e69de4142b2d92150ce030a22" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 43633 } } }
I20260927 05:01:11.153847 32662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22 [sys.catalog]: This master's current role is: LEADER
I20260927 05:01:11.156889 32671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:01:11.164580 32671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:01:11.165680 32656 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:01:11.191968 32671 catalog_manager.cc:1347] Generated new cluster ID: 791ede77a5634b0ca7fcea8a253be862
I20260927 05:01:11.192178 32671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:01:11.230577 32671 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:01:11.232833 32671 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:01:11.243564 32671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22: Generated new TSK 0
I20260927 05:01:11.244841 32671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:01:12.036954 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:37659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43633
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:01:12 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:01:12.583592 32702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:01:12.584089 32702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:01:12.584340 32702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:01:12.597206 32702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:01:12.597472 32702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:01:12.597762 32702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:01:12.611227 32702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.169.84:37659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:01:12.612769 32702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:01:12.614313 32702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:01:12.640051 32712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:12.640254 32715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:12.640254 32713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:01:12.641645 32702 server_base.cc:1034] running on GCE node
I20260927 05:01:12.642443 32702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:01:12.643923 32702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:01:12.645191 32702 hybrid_clock.cc:648] HybridClock initialized: now 1790485272645061 us; error 83 us; skew 500 ppm
I20260927 05:01:12.645645 32702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:01:12 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:01:12.654224 32702 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:01:12.658104 32702 webserver.cc:460] Webserver started at http://127.20.169.65:38919/ using document root <none> and password file <none>
I20260927 05:01:12.659970 32702 fs_manager.cc:359] Metadata directory not provided
I20260927 05:01:12.660280 32702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:01:12.660805 32702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:01:12 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:12.702293 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "2fd451d73883437e8125929fe8030d8d"
format_stamp: "Formatted at 2026-09-27 05:01:12 on dist-test-slave-39f2"
server_key: "a0f42a39dfdd6a4fa65e4cf9158842e3"
server_key_iv: "8e7019d08b4928792683ce5bb37b99c8"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:12.703877 32702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "2fd451d73883437e8125929fe8030d8d"
format_stamp: "Formatted at 2026-09-27 05:01:12 on dist-test-slave-39f2"
server_key: "a0f42a39dfdd6a4fa65e4cf9158842e3"
server_key_iv: "8e7019d08b4928792683ce5bb37b99c8"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:12.717607 32702 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.002s sys 0.009s
I20260927 05:01:12.751075 32722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:12.753451 32702 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.003s
I20260927 05:01:12.754182 32702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "2fd451d73883437e8125929fe8030d8d"
format_stamp: "Formatted at 2026-09-27 05:01:12 on dist-test-slave-39f2"
server_key: "a0f42a39dfdd6a4fa65e4cf9158842e3"
server_key_iv: "8e7019d08b4928792683ce5bb37b99c8"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:12.754642 32702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:01:12.822201 32702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:01:12.830803 32702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:01:12.831416 32702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:01:12.849619 32702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:01:12.865233 32702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:01:12.865412 32702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:12.865674 32702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:01:12.865787 32702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:01:12 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:13.187718 366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:36907 every 8 connection(s)
I20260927 05:01:13.188088 32702 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:36907
I20260927 05:01:13.189857 32702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:01:13.196460 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 32702
I20260927 05:01:13.196676 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:01:13.197631 21157 external_mini_cluster.cc:1428] Setting key 8ade0013f5f740658c7466d33fa268c9
Sep 27 05:01:13 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 05:01:13.244397 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:13.197063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:57861 (local address 127.20.169.126:43633)
0927 05:01:13.205274 (+ 8211us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:13.205283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:13.211495 (+ 6212us) server_negotiation.cc:407] Connection header received
0927 05:01:13.214074 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:13.214079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:13.214171 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:13.214283 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:13.222930 (+ 8647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:13.224105 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:13.226794 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:13.227453 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:13.227572 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:13.232230 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:13.232237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:13.232242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:13.232297 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:13.237083 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:13.241000 (+ 3917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:13.241005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:13.241023 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:13.241199 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:13.241871 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:13.241876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:13.241880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:13.242290 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:13.242451 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:13.243695 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0927 05:01:13.243987 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8028,"thread_start_us":7877,"threads_started":1}
I20260927 05:01:13.247087 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:13.197658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43633 (local address 127.20.169.65:57861)
0927 05:01:13.211322 (+ 13664us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:13.211383 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:13.213750 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:13.216665 (+ 2915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:13.216678 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:13.221629 (+ 4951us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:01:13.222620 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:13.222650 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:13.224370 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:13.224375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:13.226589 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:13.226603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:13.227911 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:13.227918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:13.228192 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:13.229058 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:01:13.229087 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:01:13.231925 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:01:13.237233 (+ 5308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:01:13.237243 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:01:13.237263 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:01:13.238079 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:01:13.241384 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:01:13.241391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:01:13.241395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:01:13.241721 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:01:13.242566 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:01:13.242575 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:01:13.243157 (+ 582us) client_negotiation.cc:769] Sending connection context
0927 05:01:13.244775 (+ 1618us) client_negotiation.cc:240] Negotiation successful
0927 05:01:13.245475 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13372,"thread_start_us":12286,"threads_started":1}
I20260927 05:01:13.249311 367 heartbeater.cc:347] Connected to a master server at 127.20.169.126:43633
I20260927 05:01:13.249720 367 heartbeater.cc:464] Registering TS with master...
I20260927 05:01:13.251155 367 heartbeater.cc:511] Master 127.20.169.126:43633 requested a full tablet report, sending...
I20260927 05:01:13.253996 32621 ts_manager.cc:194] Registered new tserver with Master: 2fd451d73883437e8125929fe8030d8d (127.20.169.65:36907)
I20260927 05:01:13.256489 32621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:57861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:01:13.280674 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:37659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43633
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:01:13 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
W20260927 05:01:13.888599 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:01:13.889062 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:01:13.889282 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:01:13.899261 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:01:13.899534 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:01:13.899827 373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:01:13.913481 373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.169.84:37659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:01:13.914904 373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:01:13.916352 373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:01:13.936614 391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:13.936570 389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:13.936904 388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:01:13.942453 373 server_base.cc:1034] running on GCE node
I20260927 05:01:13.943226 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:01:13.944342 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:01:13.945662 373 hybrid_clock.cc:648] HybridClock initialized: now 1790485273945561 us; error 89 us; skew 500 ppm
I20260927 05:01:13.946039 373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:01:13 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485273, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:01:13.949411 373 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:01:13.951172 373 webserver.cc:460] Webserver started at http://127.20.169.66:41713/ using document root <none> and password file <none>
I20260927 05:01:13.951910 373 fs_manager.cc:359] Metadata directory not provided
I20260927 05:01:13.952060 373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:01:13.952519 373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:01:13 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485273, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:13.971253 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "aeb7db4350464ed0b91c547ece4b1f18"
format_stamp: "Formatted at 2026-09-27 05:01:13 on dist-test-slave-39f2"
server_key: "13e577c0fb90e97a0c5d45fa78d8c376"
server_key_iv: "6ded5beabeca4b948b85571a9ee88b25"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:13.972321 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "aeb7db4350464ed0b91c547ece4b1f18"
format_stamp: "Formatted at 2026-09-27 05:01:13 on dist-test-slave-39f2"
server_key: "13e577c0fb90e97a0c5d45fa78d8c376"
server_key_iv: "6ded5beabeca4b948b85571a9ee88b25"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:13.979712 373 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260927 05:01:14.005511 398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:14.007637 373 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.001s
I20260927 05:01:14.007838 373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "aeb7db4350464ed0b91c547ece4b1f18"
format_stamp: "Formatted at 2026-09-27 05:01:13 on dist-test-slave-39f2"
server_key: "13e577c0fb90e97a0c5d45fa78d8c376"
server_key_iv: "6ded5beabeca4b948b85571a9ee88b25"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:14.008144 373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:01:14.078055 373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:01:14.083315 373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:01:14.083654 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:01:14.084964 373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:01:14.086706 373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:01:14.086794 373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:14.086902 373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:01:14.086956 373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:14.131875 373 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:37431
I20260927 05:01:14.131965 510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:37431 every 8 connection(s)
I20260927 05:01:14.133256 373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:01:14.141577 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 373
I20260927 05:01:14.141757 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:01:14.142328 21157 external_mini_cluster.cc:1428] Setting key 39cf5dead1bac35026776fd052f2e95c
Sep 27 05:01:14 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485273, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:01:14.176961 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:14.136206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:53133 (local address 127.20.169.126:43633)
0927 05:01:14.153248 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:14.153253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:14.153939 (+ 686us) server_negotiation.cc:407] Connection header received
0927 05:01:14.156779 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:14.156785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:14.156920 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:14.157061 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:14.159605 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.160891 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.163799 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.164507 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.164641 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:14.169209 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:14.169214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:14.169218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:14.169251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:14.173150 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:14.173979 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:14.173986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:14.173990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:14.174160 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:14.174681 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:14.174687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:14.174690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:14.175048 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:14.175200 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:14.176042 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 05:01:14.176282 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16910,"thread_start_us":16779,"threads_started":1}
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 05:01:14.178157 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:14.137093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43633 (local address 127.20.169.66:53133)
0927 05:01:14.153748 (+ 16655us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:14.153793 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:14.156427 (+ 2634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:14.157399 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:14.157412 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:14.158300 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:01:14.159401 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:14.159425 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.161062 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.161068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:14.163542 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:14.163641 (+ 99us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.164894 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.164952 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:14.165224 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:14.166148 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:01:14.166171 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:01:14.168939 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:01:14.173309 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:01:14.173319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:01:14.173338 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:01:14.173855 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:01:14.174283 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:01:14.174288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:01:14.174292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:01:14.174547 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:01:14.175189 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:01:14.175197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:01:14.175591 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 05:01:14.175885 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 05:01:14.176528 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16379,"thread_start_us":16151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 05:01:14.180491 511 heartbeater.cc:347] Connected to a master server at 127.20.169.126:43633
I20260927 05:01:14.180835 511 heartbeater.cc:464] Registering TS with master...
I20260927 05:01:14.182008 511 heartbeater.cc:511] Master 127.20.169.126:43633 requested a full tablet report, sending...
I20260927 05:01:14.184010 32621 ts_manager.cc:194] Registered new tserver with Master: aeb7db4350464ed0b91c547ece4b1f18 (127.20.169.66:37431)
I20260927 05:01:14.185539 32621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:53133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:01:14.217373 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:37659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43633
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:01:14.260676 367 heartbeater.cc:503] Master 127.20.169.126:43633 was elected leader, sending a full tablet report...
W20260927 05:01:14.731420 517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:01:14.731751 517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:01:14.731942 517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:01:14.739845 517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:01:14.739995 517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:01:14.740170 517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:01:14.749995 517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.169.84:37659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:01:14.751302 517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:01:14.752631 517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:01:14.769033 522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:14.769435 523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:01:14.769555 525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:01:14.771036 517 server_base.cc:1034] running on GCE node
I20260927 05:01:14.771781 517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:01:14.772974 517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:01:14.774261 517 hybrid_clock.cc:648] HybridClock initialized: now 1790485274774147 us; error 59 us; skew 500 ppm
I20260927 05:01:14.774694 517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:01:14 dist-test-slave-39f2 krb5kdc[32141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485274, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:01:14.779055 517 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:01:14.781147 517 webserver.cc:460] Webserver started at http://127.20.169.67:38077/ using document root <none> and password file <none>
I20260927 05:01:14.782040 517 fs_manager.cc:359] Metadata directory not provided
I20260927 05:01:14.782245 517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:01:14.782630 517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:01:14 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485274, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:14.801644 517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "bfb198ca89814144a367f01b2aec6374"
format_stamp: "Formatted at 2026-09-27 05:01:14 on dist-test-slave-39f2"
server_key: "2aedb3e4afb5fb9a45cc9939df470f39"
server_key_iv: "89df89c433a88f6839298d84f9e94155"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:14.802865 517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "bfb198ca89814144a367f01b2aec6374"
format_stamp: "Formatted at 2026-09-27 05:01:14 on dist-test-slave-39f2"
server_key: "2aedb3e4afb5fb9a45cc9939df470f39"
server_key_iv: "89df89c433a88f6839298d84f9e94155"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:14.811688 517 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260927 05:01:14.830835 533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:14.833117 517 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260927 05:01:14.833374 517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "bfb198ca89814144a367f01b2aec6374"
format_stamp: "Formatted at 2026-09-27 05:01:14 on dist-test-slave-39f2"
server_key: "2aedb3e4afb5fb9a45cc9939df470f39"
server_key_iv: "89df89c433a88f6839298d84f9e94155"
server_key_version: "kuduclusterkey@0"
I20260927 05:01:14.833752 517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:01:14.854552 517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:01:14.859675 517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:01:14.860080 517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:01:14.861531 517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:01:14.863575 517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:01:14.863713 517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:14.863962 517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:01:14.864094 517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:01:14.923728 517 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:46237
I20260927 05:01:14.923774 645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:46237 every 8 connection(s)
I20260927 05:01:14.925087 517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:01:14.931624 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 517
I20260927 05:01:14.931821 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:01:14.932374 21157 external_mini_cluster.cc:1428] Setting key 00c799ce859fd1b06fe6b313f56d2513
Sep 27 05:01:14 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485274, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:01:14.951179 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:14.927980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:42193 (local address 127.20.169.126:43633)
0927 05:01:14.928652 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:14.928656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:14.930145 (+ 1489us) server_negotiation.cc:407] Connection header received
0927 05:01:14.932637 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:14.932643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:14.932780 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:14.932957 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:14.934957 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.935951 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.938079 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.938662 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.938802 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:14.943411 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:14.943417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:14.943421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:14.943456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:14.947114 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:14.948095 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:14.948099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:14.948102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:14.948307 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:14.948955 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:14.948959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:14.948962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:14.949305 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:14.949489 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:14.950290 (+ 801us) server_negotiation.cc:299] Negotiation successful
0927 05:01:14.950623 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":329,"threads_started":1}
I20260927 05:01:14.952654 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:14.928737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:43633 (local address 127.20.169.67:42193)
0927 05:01:14.929943 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:14.930017 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:14.932335 (+ 2318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:14.933272 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:14.933280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:14.933822 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:01:14.934646 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:14.934664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.936152 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.936156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:14.937854 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:14.937884 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.938983 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.939000 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:14.939323 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:14.940142 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:01:14.940167 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:01:14.943119 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:01:14.947292 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:01:14.947302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:01:14.947322 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:01:14.947923 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:01:14.948445 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:01:14.948450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:01:14.948461 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:01:14.948763 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:01:14.949483 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:01:14.949492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:01:14.949870 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 05:01:14.950197 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 05:01:14.950905 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":634,"threads_started":1}
I20260927 05:01:14.954505 646 heartbeater.cc:347] Connected to a master server at 127.20.169.126:43633
I20260927 05:01:14.954865 646 heartbeater.cc:464] Registering TS with master...
I20260927 05:01:14.956162 646 heartbeater.cc:511] Master 127.20.169.126:43633 requested a full tablet report, sending...
I20260927 05:01:14.958693 32621 ts_manager.cc:194] Registered new tserver with Master: bfb198ca89814144a367f01b2aec6374 (127.20.169.67:46237)
I20260927 05:01:14.960217 32621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:42193
I20260927 05:01:14.961760 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:01:14.986341 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:14.969001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.20.169.126:43633)
0927 05:01:14.969160 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:14.969164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:14.969915 (+ 751us) server_negotiation.cc:407] Connection header received
0927 05:01:14.970088 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:14.970092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:14.970189 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:14.970277 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:14.972311 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.973384 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.975289 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:14.975982 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:14.976095 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:14.979274 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:14.979280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:14.979285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:14.979322 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:14.982952 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:14.983823 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:14.983827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:14.983829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:14.984017 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:14.984465 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:14.984468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:14.984471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:14.984942 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:14.985217 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:14.985569 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 05:01:14.985801 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:01:14.991241 32621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:01:15.000877 32621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:01:15.039825 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.018089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.20.169.67:46237)
0927 05:01:15.025474 (+ 7385us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.025495 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.025565 (+ 70us) server_negotiation.cc:407] Connection header received
0927 05:01:15.025725 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.025734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.026088 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.026385 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:01:15.030130 (+ 3745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.032899 (+ 2769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.036261 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.037884 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.037998 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.038246 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.039180 (+ 934us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.039336 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7039,"thread_start_us":6626,"threads_started":1}
I20260927 05:01:15.039861 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.018990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.20.169.65:36907)
0927 05:01:15.024220 (+ 5230us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.024233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.026133 (+ 1900us) server_negotiation.cc:407] Connection header received
0927 05:01:15.026904 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.026924 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.027348 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.027577 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:01:15.029539 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.032234 (+ 2695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.035765 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.037662 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.037832 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.038198 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.039068 (+ 870us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.039281 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4941,"thread_start_us":4677,"threads_started":1}
I20260927 05:01:15.039929 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.017851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:46237 (local address 127.0.0.1:37978)
0927 05:01:15.020098 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:15.020137 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:15.020352 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:15.026362 (+ 6010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:15.026370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:15.026403 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:01:15.029697 (+ 3294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.029730 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.033092 (+ 3362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.033098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.036053 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.036068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.038556 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.038563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.038916 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.038957 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 05:01:15.039061 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:01:15.039442 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"mutex_wait_us":128,"thread_start_us":7460,"threads_started":1}
I20260927 05:01:15.040567 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.018569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36907 (local address 127.0.0.1:34078)
0927 05:01:15.025793 (+ 7224us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:15.025814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:15.025981 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:15.028063 (+ 2082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:15.028068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:15.028092 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:01:15.029180 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.029198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.032420 (+ 3222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.032426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.035605 (+ 3179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.035620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.038463 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.038476 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.038819 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.038861 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:01:15.039006 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 05:01:15.040156 (+ 1150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7032,"mutex_wait_us":1141,"thread_start_us":926,"threads_started":1}
I20260927 05:01:15.044188 663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.025842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:37431 (local address 127.0.0.1:56782)
0927 05:01:15.034015 (+ 8173us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:15.034041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:15.034170 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:15.035728 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:15.035734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:15.035759 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:01:15.036253 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.036264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.037901 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.037906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.041547 (+ 3641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.041560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.043300 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.043307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.043490 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.043528 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:01:15.043643 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 05:01:15.043772 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8013,"mutex_wait_us":32,"thread_start_us":9673,"threads_started":1}
I20260927 05:01:15.044230 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.026340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.20.169.66:37431)
0927 05:01:15.034666 (+ 8326us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.034676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.034708 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:01:15.034823 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.034833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.035092 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.035321 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:01:15.036447 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.037696 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.041703 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.043007 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.043187 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.043412 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.043698 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.043875 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8159,"thread_start_us":8011,"threads_started":1}
I20260927 05:01:15.044369 581 tablet_service.cc:1467] Processing CreateTablet for tablet 081de575db0348f2be5c1534ad81aacb (DEFAULT_TABLE table=test-table [id=74da474d84d046e39bfb242b464dc362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:01:15.044369 302 tablet_service.cc:1467] Processing CreateTablet for tablet 081de575db0348f2be5c1534ad81aacb (DEFAULT_TABLE table=test-table [id=74da474d84d046e39bfb242b464dc362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:01:15.045979 302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081de575db0348f2be5c1534ad81aacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:01:15.046204 581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081de575db0348f2be5c1534ad81aacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:01:15.048133 446 tablet_service.cc:1467] Processing CreateTablet for tablet 081de575db0348f2be5c1534ad81aacb (DEFAULT_TABLE table=test-table [id=74da474d84d046e39bfb242b464dc362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:01:15.049446 446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081de575db0348f2be5c1534ad81aacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:01:15.066140 667 tablet_bootstrap.cc:492] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Bootstrap starting.
I20260927 05:01:15.066295 665 tablet_bootstrap.cc:492] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Bootstrap starting.
I20260927 05:01:15.070387 667 tablet_bootstrap.cc:654] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Neither blocks nor log segments found. Creating new log.
I20260927 05:01:15.070775 666 tablet_bootstrap.cc:492] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Bootstrap starting.
I20260927 05:01:15.071247 665 tablet_bootstrap.cc:654] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Neither blocks nor log segments found. Creating new log.
I20260927 05:01:15.072763 667 log.cc:824] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Log is configured to *not* fsync() on all Append() calls
I20260927 05:01:15.073931 665 log.cc:824] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:01:15.080422 666 tablet_bootstrap.cc:654] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Neither blocks nor log segments found. Creating new log.
I20260927 05:01:15.083101 666 log.cc:824] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Log is configured to *not* fsync() on all Append() calls
I20260927 05:01:15.084645 667 tablet_bootstrap.cc:492] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: No bootstrap required, opened a new log
I20260927 05:01:15.085230 667 ts_tablet_manager.cc:1397] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20260927 05:01:15.101425 665 tablet_bootstrap.cc:492] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: No bootstrap required, opened a new log
I20260927 05:01:15.103657 665 ts_tablet_manager.cc:1397] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.015s
I20260927 05:01:15.103637 667 raft_consensus.cc:348] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.104348 667 raft_consensus.cc:374] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:01:15.104526 667 raft_consensus.cc:729] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb7db4350464ed0b91c547ece4b1f18, State: Initialized, Role: FOLLOWER
I20260927 05:01:15.105607 666 tablet_bootstrap.cc:492] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: No bootstrap required, opened a new log
I20260927 05:01:15.105561 667 consensus_queue.cc:260] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.106273 666 ts_tablet_manager.cc:1397] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.023s
I20260927 05:01:15.109301 511 heartbeater.cc:503] Master 127.20.169.126:43633 was elected leader, sending a full tablet report...
I20260927 05:01:15.112669 667 ts_tablet_manager.cc:1428] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Time spent starting tablet: real 0.027s user 0.015s sys 0.008s
I20260927 05:01:15.119190 665 raft_consensus.cc:348] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.119619 665 raft_consensus.cc:374] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:01:15.119737 665 raft_consensus.cc:729] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd451d73883437e8125929fe8030d8d, State: Initialized, Role: FOLLOWER
I20260927 05:01:15.119534 666 raft_consensus.cc:348] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.120035 666 raft_consensus.cc:374] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:01:15.120169 666 raft_consensus.cc:729] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb198ca89814144a367f01b2aec6374, State: Initialized, Role: FOLLOWER
I20260927 05:01:15.120499 665 consensus_queue.cc:260] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.120738 666 consensus_queue.cc:260] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.122653 666 ts_tablet_manager.cc:1428] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Time spent starting tablet: real 0.016s user 0.006s sys 0.009s
I20260927 05:01:15.123154 646 heartbeater.cc:503] Master 127.20.169.126:43633 was elected leader, sending a full tablet report...
I20260927 05:01:15.124800 665 ts_tablet_manager.cc:1428] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Time spent starting tablet: real 0.021s user 0.016s sys 0.000s
W20260927 05:01:15.146450 512 tablet.cc:2369] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:01:15.177153 647 tablet.cc:2369] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:01:15.216358 370 tablet.cc:2369] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:01:15.344494 674 raft_consensus.cc:482] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:01:15.344866 674 raft_consensus.cc:504] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.346697 674 leader_election.cc:288] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fd451d73883437e8125929fe8030d8d (127.20.169.65:36907), aeb7db4350464ed0b91c547ece4b1f18 (127.20.169.66:37431)
I20260927 05:01:15.354429 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.347153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:60217 (local address 127.20.169.65:36907)
0927 05:01:15.347421 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.347428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.347701 (+ 273us) server_negotiation.cc:407] Connection header received
0927 05:01:15.347958 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.347962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.348026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.348121 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:01:15.348496 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.349607 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.352114 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.353432 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.353629 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.353720 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.354005 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.354107 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 05:01:15.354419 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.347157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36907 (local address 127.20.169.67:60217)
0927 05:01:15.347570 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:15.347593 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:15.347771 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:15.348153 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:15.348158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:15.348184 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:01:15.348350 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.348358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.349761 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.349766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.351932 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.351946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.353608 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.353613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.353815 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.353864 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 05:01:15.353976 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 05:01:15.354110 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":77}
I20260927 05:01:15.354748 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.347153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:58779 (local address 127.20.169.66:37431)
0927 05:01:15.347401 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.347408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.348026 (+ 618us) server_negotiation.cc:407] Connection header received
0927 05:01:15.348243 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.348248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.348358 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.348473 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:01:15.348982 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.349958 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.352337 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.353664 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.353839 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.353891 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.354319 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.354459 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:01:15.354874 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.347162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:37431 (local address 127.20.169.67:58779)
0927 05:01:15.347941 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 05:01:15.347954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 05:01:15.348077 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:01:15.348485 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:01:15.348489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:01:15.348512 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:01:15.348786 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.348793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.350115 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.350120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.352174 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:01:15.352183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.353868 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.353873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:01:15.354044 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.354085 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 05:01:15.354229 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:01:15.354365 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":76,"thread_start_us":354,"threads_started":1}
I20260927 05:01:15.355837 322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081de575db0348f2be5c1534ad81aacb" candidate_uuid: "bfb198ca89814144a367f01b2aec6374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd451d73883437e8125929fe8030d8d" is_pre_election: true
I20260927 05:01:15.356003 466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081de575db0348f2be5c1534ad81aacb" candidate_uuid: "bfb198ca89814144a367f01b2aec6374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb7db4350464ed0b91c547ece4b1f18" is_pre_election: true
I20260927 05:01:15.356405 322 raft_consensus.cc:2454] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb198ca89814144a367f01b2aec6374 in term 0.
I20260927 05:01:15.356617 466 raft_consensus.cc:2454] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb198ca89814144a367f01b2aec6374 in term 0.
I20260927 05:01:15.357270 534 leader_election.cc:302] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd451d73883437e8125929fe8030d8d, bfb198ca89814144a367f01b2aec6374; no voters:
I20260927 05:01:15.357753 674 raft_consensus.cc:2781] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:01:15.357896 674 raft_consensus.cc:482] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:01:15.358022 674 raft_consensus.cc:3037] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:01:15.360272 674 raft_consensus.cc:504] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.361037 674 leader_election.cc:288] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [CANDIDATE]: Term 1 election: Requested vote from peers 2fd451d73883437e8125929fe8030d8d (127.20.169.65:36907), aeb7db4350464ed0b91c547ece4b1f18 (127.20.169.66:37431)
I20260927 05:01:15.361419 322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081de575db0348f2be5c1534ad81aacb" candidate_uuid: "bfb198ca89814144a367f01b2aec6374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd451d73883437e8125929fe8030d8d"
I20260927 05:01:15.361609 322 raft_consensus.cc:3037] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:01:15.361596 466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081de575db0348f2be5c1534ad81aacb" candidate_uuid: "bfb198ca89814144a367f01b2aec6374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb7db4350464ed0b91c547ece4b1f18"
I20260927 05:01:15.361820 466 raft_consensus.cc:3037] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:01:15.363864 322 raft_consensus.cc:2454] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb198ca89814144a367f01b2aec6374 in term 1.
I20260927 05:01:15.364291 534 leader_election.cc:302] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd451d73883437e8125929fe8030d8d, bfb198ca89814144a367f01b2aec6374; no voters:
I20260927 05:01:15.364283 466 raft_consensus.cc:2454] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb198ca89814144a367f01b2aec6374 in term 1.
I20260927 05:01:15.364624 674 raft_consensus.cc:2781] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:01:15.365176 674 raft_consensus.cc:686] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 1 LEADER]: Becoming Leader. State: Replica: bfb198ca89814144a367f01b2aec6374, State: Running, Role: LEADER
I20260927 05:01:15.365793 674 consensus_queue.cc:237] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } }
I20260927 05:01:15.370591 32620 catalog_manager.cc:5495] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb198ca89814144a367f01b2aec6374 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "bfb198ca89814144a367f01b2aec6374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb198ca89814144a367f01b2aec6374" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 } health_report { overall_health: UNKNOWN } } }
I20260927 05:01:15.402822 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.397175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.20.169.67:46237)
0927 05:01:15.397430 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.397434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.397518 (+ 84us) server_negotiation.cc:407] Connection header received
0927 05:01:15.397737 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.397741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.397850 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.397992 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:01:15.398391 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.399084 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.400650 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.401115 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.401216 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.401729 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:01:15.402229 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:01:15.402338 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.402442 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.402508 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 05:01:15.411230 322 raft_consensus.cc:1261] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 1 FOLLOWER]: Refusing update from remote peer bfb198ca89814144a367f01b2aec6374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:01:15.411230 466 raft_consensus.cc:1261] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 1 FOLLOWER]: Refusing update from remote peer bfb198ca89814144a367f01b2aec6374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:01:15.412176 676 consensus_queue.cc:1035] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fd451d73883437e8125929fe8030d8d" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:01:15.412547 674 consensus_queue.cc:1035] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb7db4350464ed0b91c547ece4b1f18" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:01:15.433921 679 mvcc.cc:205] Tried to move back new op lower bound from 7333827688073469952 to 7333827687902576640. Current Snapshot: MvccSnapshot[applied={T|T < 7333827688073469952}]
I20260927 05:01:15.441000 680 mvcc.cc:205] Tried to move back new op lower bound from 7333827688073469952 to 7333827687902576640. Current Snapshot: MvccSnapshot[applied={T|T < 7333827688073469952}]
I20260927 05:01:15.453372 32620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:01:15.453697 32620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:01:15.457834 32620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c78be62e69de4142b2d92150ce030a22: Sending DeleteTablet for 3 replicas of tablet 081de575db0348f2be5c1534ad81aacb
I20260927 05:01:15.459064 302 tablet_service.cc:1514] Processing DeleteTablet for tablet 081de575db0348f2be5c1534ad81aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:01:15 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34078
I20260927 05:01:15.459088 301 tablet_service.cc:1514] Processing DeleteTablet for tablet 081de575db0348f2be5c1534ad81aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:01:15 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:34078
I20260927 05:01:15.460544 446 tablet_service.cc:1514] Processing DeleteTablet for tablet 081de575db0348f2be5c1534ad81aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:01:15 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:56782
I20260927 05:01:15.461956 581 tablet_service.cc:1514] Processing DeleteTablet for tablet 081de575db0348f2be5c1534ad81aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:01:15 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:37978
I20260927 05:01:15.464493 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 32702
I20260927 05:01:15.471655 690 tablet_replica.cc:330] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: stopping tablet replica
I20260927 05:01:15.472364 690 raft_consensus.cc:2229] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:01:15.473304 690 raft_consensus.cc:2258] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:01:15.475600 690 ts_tablet_manager.cc:1905] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:01:15.484308 690 ts_tablet_manager.cc:1918] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:01:15.485119 689 tablet_replica.cc:330] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: stopping tablet replica
I20260927 05:01:15.485769 689 raft_consensus.cc:2229] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:01:15.486492 689 raft_consensus.cc:2258] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:01:15.487282 690 log.cc:1196] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/081de575db0348f2be5c1534ad81aacb
I20260927 05:01:15.487977 690 ts_tablet_manager.cc:1939] T 081de575db0348f2be5c1534ad81aacb P aeb7db4350464ed0b91c547ece4b1f18: Deleting consensus metadata
I20260927 05:01:15.489168 689 ts_tablet_manager.cc:1905] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:01:15.490388 32606 catalog_manager.cc:4841] TS aeb7db4350464ed0b91c547ece4b1f18 (127.20.169.66:37431): tablet 081de575db0348f2be5c1534ad81aacb (table test-table [id=74da474d84d046e39bfb242b464dc362]) successfully deleted
I20260927 05:01:15.492076 691 tablet_replica.cc:330] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: stopping tablet replica
I20260927 05:01:15.492575 691 raft_consensus.cc:2229] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:01:15.493382 691 raft_consensus.cc:2258] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:01:15.496068 691 ts_tablet_manager.cc:1905] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:01:15 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:01:15.508224 689 ts_tablet_manager.cc:1918] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:01:15.508415 689 log.cc:1196] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/081de575db0348f2be5c1534ad81aacb
I20260927 05:01:15.509145 689 ts_tablet_manager.cc:1939] T 081de575db0348f2be5c1534ad81aacb P 2fd451d73883437e8125929fe8030d8d: Deleting consensus metadata
I20260927 05:01:15.511818 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.467891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.20.169.65:36907)
0927 05:01:15.468120 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.468129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.491403 (+ 23274us) server_negotiation.cc:407] Connection header received
0927 05:01:15.491627 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.491634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.491743 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.491856 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:15.493837 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.495040 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.497202 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.497826 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.497928 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.503599 (+ 5671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:15.503617 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:15.503641 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:15.503682 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:15.508013 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:15.509036 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:15.509046 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:15.509049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:15.509230 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:15.509690 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:15.509695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:15.509699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:15.510231 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:15.510559 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.511158 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.511393 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:01:15.511829 691 ts_tablet_manager.cc:1918] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:01:15.512006 691 log.cc:1196] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/081de575db0348f2be5c1534ad81aacb
I20260927 05:01:15.512614 691 ts_tablet_manager.cc:1939] T 081de575db0348f2be5c1534ad81aacb P bfb198ca89814144a367f01b2aec6374: Deleting consensus metadata
I20260927 05:01:15.513363 362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:01:15.513700 32604 catalog_manager.cc:4841] TS 2fd451d73883437e8125929fe8030d8d (127.20.169.65:36907): tablet 081de575db0348f2be5c1534ad81aacb (table test-table [id=74da474d84d046e39bfb242b464dc362]) successfully deleted
W20260927 05:01:15.514293 32604 catalog_manager.cc:4808] TS 2fd451d73883437e8125929fe8030d8d (127.20.169.65:36907): delete failed for tablet 081de575db0348f2be5c1534ad81aacb because the tablet was not found. No further retry: Not found: Tablet not found: 081de575db0348f2be5c1534ad81aacb
I20260927 05:01:15.515877 32606 catalog_manager.cc:4841] TS bfb198ca89814144a367f01b2aec6374 (127.20.169.67:46237): tablet 081de575db0348f2be5c1534ad81aacb (table test-table [id=74da474d84d046e39bfb242b464dc362]) successfully deleted
I20260927 05:01:15.606535 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 32702
I20260927 05:01:15.625521 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 373
Sep 27 05:01:15 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:01:15.644323 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.626547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.20.169.66:37431)
0927 05:01:15.626734 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.626740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.626760 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:01:15.626830 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.626835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.626903 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.627009 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:15.628720 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.629603 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.631593 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.632024 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.632132 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.637759 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:15.637775 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:15.637791 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:15.637834 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:15.641220 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:15.642104 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:15.642111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:15.642113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:15.642253 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:15.642726 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:15.642731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:15.642735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:15.643144 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:15.643369 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.643776 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.643945 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:01:15.645159 506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:01:15.734141 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 373
I20260927 05:01:15.757982 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 517
Sep 27 05:01:15 dist-test-slave-39f2 krb5kdc[32141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:01:15.776312 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:01:15.759062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.20.169.67:46237)
0927 05:01:15.759300 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:01:15.759306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:01:15.759329 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:01:15.759397 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:01:15.759403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:01:15.759502 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:01:15.759631 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:01:15.761157 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.762418 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.763962 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:01:15.764665 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:01:15.764985 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:01:15.769359 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:01:15.769374 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:01:15.769399 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:01:15.769442 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:01:15.773410 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:15.774038 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:15.774052 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:15.774056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:15.774251 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:01:15.774593 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:01:15.774597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:01:15.774603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:01:15.775075 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:01:15.775408 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 05:01:15.775638 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 05:01:15.775898 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:01:15.777148 641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:01:15.898588 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 517
I20260927 05:01:15.920011 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 32588
I20260927 05:01:15.921310 32651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:01:16.037878 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 32588
I20260927 05:01:16.063884 21157 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 05:01:16.399117 21157 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 05:01:16.399325 21157 mini_ranger.cc:66] Stopping Ranger...
I20260927 05:01:16.745043 21157 mini_ranger.cc:68] Stopped Ranger
2026-09-27 05:01:16.745 UTC [32171] LOG: received smart shutdown request
2026-09-27 05:01:16.748 UTC [32171] LOG: background worker "logical replication launcher" (PID 32180) exited with exit code 1
2026-09-27 05:01:16.749 UTC [32175] LOG: shutting down
2026-09-27 05:01:16.917 UTC [32171] LOG: database system is shut down
2026-09-27T05:01:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50629 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:01:17 dist-test-slave-39f2 krb5kdc[705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:01:17 dist-test-slave-39f2 krb5kdc[705](info): set up 2 sockets
Sep 27 05:01:17 dist-test-slave-39f2 krb5kdc[705](info): commencing operation
krb5kdc: starting...
W20260927 05:01:18.185216 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:01:18 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:01:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/rangeradmin_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/rangeradmin_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/rangerlookup_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/rangerlookup_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
I20260927 05:01:18.515884 21157 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRTYp1d/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/postgres -l logfile start
2026-09-27 05:01:20.977 UTC [735] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 05:01:20.977 UTC [735] LOG: listening on IPv4 address "127.20.169.84", port 60949
2026-09-27 05:01:20.981 UTC [735] LOG: listening on Unix socket "/tmp/.s.PGSQL.60949"
2026-09-27 05:01:21.006 UTC [738] LOG: database system was shut down at 2026-09-27 05:01:19 UTC
2026-09-27 05:01:21.010 UTC [735] LOG: database system is ready to accept connections
I20260927 05:01:22.004956 21157 mini_postgres.cc:96] Postgres bound to 60949
2026-09-27 05:01:22.023 UTC [746] FATAL: database "slave" does not exist
127.20.169.84:60949 - accepting connections
I20260927 05:01:22.024185 21157 mini_ranger.cc:161] Starting Ranger...
I20260927 05:01:22.050596 21157 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 05:01:22.532948 21157 mini_ranger.cc:87] Created ranger Postgres database
I20260927 05:01:22.533037 21157 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-admin
2026-09-27 05:01:22,827 [I] DB FLAVOR :POSTGRES
2026-09-27 05:01:22,828 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:01:22,828 [I] Checking connection..
2026-09-27 05:01:22,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 05:01:23,089 [I] Checking connection passed.
2026-09-27 05:01:23,089 [I] --------- Verifying version history table ---------
2026-09-27 05:01:23,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:01:23,336 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:01:23,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:01:23,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:01:23,597 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 05:01:23,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 05:01:23.874 UTC [810] WARNING: there is no transaction in progress
2026-09-27 05:01:23,880 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 05:01:23,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:01:24,129 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 05:01:24,130 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:01:24,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 05:01:24,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 05:01:24,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-39f2.c.gcp-upstream.internal','N') ;"
2026-09-27 05:01:24,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:01:25,098 [I] Table x_portal_user does not exist in database ranger
2026-09-27 05:01:25,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:01:25,352 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 05:01:25,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:01:25,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:01:25,867 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 05:01:25,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 05:01:26.853 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:26.870 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:26.887 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:26.902 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:26.920 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:26.940 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.006 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.013 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.028 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.039 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.049 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.064 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.073 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.080 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.086 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27.093 UTC [920] WARNING: there is no transaction in progress
2026-09-27 05:01:27,732 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 05:01:27,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 05:01:27,962 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 05:01:27,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:01:28,216 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 05:01:28,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:01:28,451 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 05:01:28,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:01:28,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:01:28,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-39f2.c.gcp-upstream.internal';"
2026-09-27 05:01:29,204 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 05:01:29,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 05:01:29,465 [I] DB_PATCHES have already been applied
I20260927 05:01:29.471172 21157 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRTYp1d/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 05:01:29.471269 21157 mini_ranger.cc:193] Using host: 127.20.169.84
I20260927 05:01:29.486987 21157 mini_ranger.cc:227] Ranger admin URL: http://127.20.169.84:45363
Sep 27, 2026 5:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 5:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 5:01:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 5:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 5:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 05:01:30.637316 21157 mini_ranger.cc:160] Time spent starting Ranger: real 8.613s user 0.003s sys 0.035s
Sep 27, 2026 5:01:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 5:01:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 5:01:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 5:01:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 5:01:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 05:01:52.553013 21157 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.169.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.169.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/rangerkms_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.169.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/rangerkms_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.169.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.169.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.169.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/HTTP_127.20.169.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 05:01:52 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 05:01:52.724750 21157 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 05:01:52.746680 21157 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 05:01:53.585249 21157 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 05:01:53.585390 21157 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms
I20260927 05:01:53.585440 21157 mini_ranger_kms.cc:227] Using postgres at 127.20.169.84:60949
2026-09-27 05:01:53,945 [I] DB FLAVOR :POSTGRES
2026-09-27 05:01:53,945 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:01:53,946 [I] Checking connection
2026-09-27 05:01:53,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 05:01:54,197 [I] connection success
2026-09-27 05:01:54,197 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 05:01:54,197 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 05:01:54,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 05:01:54,431 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 05:01:54,431 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:01:54,431 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 05:01:54,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.169.84:60949/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 05:01:54,689 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 05:01:55.179497 21157 mini_ranger_kms.cc:310] Created kms service
I20260927 05:01:55.401427 21157 mini_ranger_kms.cc:326] Created kudu user
I20260927 05:01:55.449110 21157 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 05:01:55 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:01:55.942013 21157 mini_ranger_kms.cc:384] Added ranger policy
I20260927 05:01:55.942333 21157 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRTYp1d/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRTYp1d/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 05:01:55.942394 21157 mini_ranger_kms.cc:240] Using host: 127.20.169.84
I20260927 05:01:55.961272 21157 mini_ranger_kms.cc:276] Ranger KMS PID: 1102
I20260927 05:01:55.961408 21157 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.169.84:43585
Sep 27, 2026 5:01:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:01:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:01:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 5:01:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 5:01:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 5:01:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 05:01:58.323560 21157 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.599s user 0.067s sys 0.315s
I20260927 05:01:58.323829 21157 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 05:01:58.324011 21157 mini_ranger_kms.cc:401] 127.20.169.84:43585/kms/v1/keys
Sep 27 05:01:58 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.169.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 05:01:58 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.169.84@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
Sep 27 05:01:59 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.169.84@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
Sep 27 05:02:01 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:02:01.424899 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39321
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:39321
--ranger_config_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:01.774559 1153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:01.774843 1153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:01.774992 1153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:01.782225 1153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:01.782325 1153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:01.782407 1153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:01.782509 1153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:01.782569 1153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:01.792557 1153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.169.84:43585/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:39321
--ranger_config_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:01.793849 1153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:01.795149 1153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:01.808490 1158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:01.808535 1161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:01.808444 1159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:01.809087 1153 server_base.cc:1034] running on GCE node
I20260927 05:02:01.809918 1153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:01.811393 1153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:01.812633 1153 hybrid_clock.cc:648] HybridClock initialized: now 1790485321812575 us; error 44 us; skew 500 ppm
I20260927 05:02:01.813062 1153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:01 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:01.816599 1153 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:01.818444 1153 webserver.cc:460] Webserver started at http://127.20.169.126:38601/ using document root <none> and password file <none>
I20260927 05:02:01.819252 1153 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:01.819381 1153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:01.819706 1153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:02:01 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:01.871338 1153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "e677807975a840fca5c98330cfd8dc8e"
format_stamp: "Formatted at 2026-09-27 05:02:01 on dist-test-slave-39f2"
server_key: "3c748cead68b323a97f2116755866edb"
server_key_iv: "b2ccaa46a39b8b3734819235d0b4c8c4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:01.872455 1153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "e677807975a840fca5c98330cfd8dc8e"
format_stamp: "Formatted at 2026-09-27 05:02:01 on dist-test-slave-39f2"
server_key: "3c748cead68b323a97f2116755866edb"
server_key_iv: "b2ccaa46a39b8b3734819235d0b4c8c4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:01.879293 1153 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260927 05:02:01.899097 1168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:01.900789 1153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 05:02:01.901086 1153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "e677807975a840fca5c98330cfd8dc8e"
format_stamp: "Formatted at 2026-09-27 05:02:01 on dist-test-slave-39f2"
server_key: "3c748cead68b323a97f2116755866edb"
server_key_iv: "b2ccaa46a39b8b3734819235d0b4c8c4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:01.901304 1153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:01.928930 1153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:01.933662 1153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:01.934034 1153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:01.956200 1153 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:39321
I20260927 05:02:01.956142 1219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:39321 every 8 connection(s)
I20260927 05:02:01.957722 1153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:01.962735 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:01.963194 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1153
I20260927 05:02:01.963474 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:02:01.964197 21157 external_mini_cluster.cc:1428] Setting key 165ea6c0fca11810bdd83b4d7fac44f1
I20260927 05:02:01.978008 1220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Bootstrap starting.
I20260927 05:02:01.982174 1220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:01.983605 1220 log.cc:824] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:01.988269 1220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: No bootstrap required, opened a new log
Sep 27 05:02:01 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:01.999729 1220 raft_consensus.cc:348] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:02.000211 1220 raft_consensus.cc:374] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:02.000324 1220 raft_consensus.cc:729] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677807975a840fca5c98330cfd8dc8e, State: Initialized, Role: FOLLOWER
I20260927 05:02:02.001101 1220 consensus_queue.cc:260] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:02.001397 1220 raft_consensus.cc:388] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:02.001543 1220 raft_consensus.cc:482] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:02.001732 1220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:02.004078 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:01.966798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.20.169.126:39321)
0927 05:02:01.981341 (+ 14543us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:01.981382 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:01.981439 (+ 57us) server_negotiation.cc:407] Connection header received
0927 05:02:01.983392 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:01.983417 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:01.983883 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:01.984326 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:01.986255 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:01.987715 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:01.989427 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:01.990381 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:01.991428 (+ 1047us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:01.995160 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:01.995169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:01.995186 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:01.995224 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:01.998577 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:01.999349 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:01.999357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:01.999364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:01.999511 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:01.999936 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:01.999946 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:01.999948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:02.001524 (+ 1576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:02.001914 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:02.002476 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 05:02:02.002972 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14250,"thread_start_us":17970,"threads_started":1}
I20260927 05:02:02.005172 1220 raft_consensus.cc:504] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:02.005803 1220 leader_election.cc:302] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e677807975a840fca5c98330cfd8dc8e; no voters:
I20260927 05:02:02.006549 1220 leader_election.cc:288] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:02:02.006767 1225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:02.011196 1225 raft_consensus.cc:686] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 LEADER]: Becoming Leader. State: Replica: e677807975a840fca5c98330cfd8dc8e, State: Running, Role: LEADER
I20260927 05:02:02.012185 1220 sys_catalog.cc:564] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:02.012547 1225 consensus_queue.cc:237] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:02.015468 1228 sys_catalog.cc:455] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e677807975a840fca5c98330cfd8dc8e. Latest consensus state: current_term: 1 leader_uuid: "e677807975a840fca5c98330cfd8dc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } } }
I20260927 05:02:02.015882 1228 sys_catalog.cc:458] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:02.016445 1226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e677807975a840fca5c98330cfd8dc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } } }
I20260927 05:02:02.016767 1226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:02.022245 1232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:02.029577 1232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:02.044477 1220 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:02:02.045761 1232 catalog_manager.cc:1347] Generated new cluster ID: 6d7c4590e4c14ac5a48fcd0acc827c5b
I20260927 05:02:02.045882 1232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:02.079239 1232 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:02:02.081938 1232 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:02.095520 1232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Generated new TSK 0
I20260927 05:02:02.096698 1232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:02:02.926468 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39321
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:02:03 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485323, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:02:03.359884 1266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:03.360334 1266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:03.360548 1266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:03.371553 1266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:03.371757 1266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:03.371994 1266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:02:03.387871 1266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.169.84:43585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:03.389896 1266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:03.391657 1266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:03.410224 1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:03.410195 1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:03.421799 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:03.422755 1266 server_base.cc:1034] running on GCE node
I20260927 05:02:03.423429 1266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:03.424573 1266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:03.425792 1266 hybrid_clock.cc:648] HybridClock initialized: now 1790485323425733 us; error 50 us; skew 500 ppm
I20260927 05:02:03.426170 1266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:03 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485323, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:03.434413 1266 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:02:03.436357 1266 webserver.cc:460] Webserver started at http://127.20.169.65:36597/ using document root <none> and password file <none>
I20260927 05:02:03.437237 1266 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:03.437387 1266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:03.437685 1266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:02:03 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485323, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:03.456954 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "04bd557fd126400e96f11db2433334c5"
format_stamp: "Formatted at 2026-09-27 05:02:03 on dist-test-slave-39f2"
server_key: "2825e359fe60a3b15eef99b52fe548c6"
server_key_iv: "7ce53db2459edf2bec865352e4d01db9"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:03.458173 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "04bd557fd126400e96f11db2433334c5"
format_stamp: "Formatted at 2026-09-27 05:02:03 on dist-test-slave-39f2"
server_key: "2825e359fe60a3b15eef99b52fe548c6"
server_key_iv: "7ce53db2459edf2bec865352e4d01db9"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:03.469805 1266 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.008s
I20260927 05:02:03.497439 1286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:03.499480 1266 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.004s
I20260927 05:02:03.499765 1266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "04bd557fd126400e96f11db2433334c5"
format_stamp: "Formatted at 2026-09-27 05:02:03 on dist-test-slave-39f2"
server_key: "2825e359fe60a3b15eef99b52fe548c6"
server_key_iv: "7ce53db2459edf2bec865352e4d01db9"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:03.500304 1266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:03.534080 1266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:03.539629 1266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:03.540069 1266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:03.546128 1266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:03.559480 1266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:03.559700 1266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 05:02:03.559937 1266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:03.560120 1266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:02:03 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485323, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:04.146667 1266 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:45899
I20260927 05:02:04.147413 1400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:45899 every 8 connection(s)
I20260927 05:02:04.148226 1266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:02:04.155277 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1266
I20260927 05:02:04.155524 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:02:04.156358 21157 external_mini_cluster.cc:1428] Setting key 020fc973d44a899b74c5b39f05cf62ec
Sep 27 05:02:04 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485323, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
Sep 27 05:02:04 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485323, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 05:02:04.192904 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:04.153180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:47137 (local address 127.20.169.126:39321)
0927 05:02:04.162791 (+ 9611us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:04.162801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:04.165024 (+ 2223us) server_negotiation.cc:407] Connection header received
0927 05:02:04.170515 (+ 5491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:04.170521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:04.170617 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:04.170691 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:04.172993 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.174151 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.176762 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.177452 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.177578 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:04.185008 (+ 7430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:04.185014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:04.185019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:04.185058 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:04.189087 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:04.190127 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:04.190135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:04.190141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:04.190294 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:04.190865 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:04.190870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:04.190873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:04.191225 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:04.191352 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:04.192306 (+ 954us) server_negotiation.cc:299] Negotiation successful
0927 05:02:04.192520 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9448,"thread_start_us":12578,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 05:02:04.195436 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:04.152487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39321 (local address 127.20.169.65:47137)
0927 05:02:04.163485 (+ 10998us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:04.163548 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:04.170138 (+ 6590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:04.171254 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:04.171267 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:04.171905 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:04.172753 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:04.172780 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.174367 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.174372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:04.176600 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:04.176616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.177943 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.177950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:04.178249 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:04.182059 (+ 3810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:04.182089 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:04.184718 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:04.189276 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:04.189287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:04.189312 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:04.189962 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:04.190460 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:04.190465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:04.190469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:04.190730 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:04.191441 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:04.191449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:04.191880 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 05:02:04.193441 (+ 1561us) client_negotiation.cc:240] Negotiation successful
0927 05:02:04.193959 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10605,"thread_start_us":12207,"threads_started":1}
I20260927 05:02:04.197496 1401 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39321
I20260927 05:02:04.197803 1401 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:04.198649 1401 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
I20260927 05:02:04.201335 1185 ts_manager.cc:194] Registered new tserver with Master: 04bd557fd126400e96f11db2433334c5 (127.20.169.65:45899)
I20260927 05:02:04.203375 1185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:47137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:02:04.237233 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39321
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:04.660102 1408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:04.660460 1408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:04.660617 1408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:04.668112 1408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:04.668275 1408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:04.668450 1408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:02:04.678375 1408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.169.84:43585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:04.679705 1408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:04.681089 1408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:04.695729 1421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:04.695804 1420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:04.696456 1423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:04.696931 1408 server_base.cc:1034] running on GCE node
I20260927 05:02:04.697507 1408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:04.698513 1408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:04.699753 1408 hybrid_clock.cc:648] HybridClock initialized: now 1790485324699702 us; error 46 us; skew 500 ppm
I20260927 05:02:04.700076 1408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:04 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485324, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:04.703316 1408 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:02:04.705015 1408 webserver.cc:460] Webserver started at http://127.20.169.66:43291/ using document root <none> and password file <none>
I20260927 05:02:04.705793 1408 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:04.705945 1408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:04.706307 1408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:02:04 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485324, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:04.724547 1408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "e7e5da45ca934ea4b879d2a082a78b49"
format_stamp: "Formatted at 2026-09-27 05:02:04 on dist-test-slave-39f2"
server_key: "29abb764a8d1298c4dd02eec2b234726"
server_key_iv: "37b14fe2c5202553325e298937921de4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:04.725620 1408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "e7e5da45ca934ea4b879d2a082a78b49"
format_stamp: "Formatted at 2026-09-27 05:02:04 on dist-test-slave-39f2"
server_key: "29abb764a8d1298c4dd02eec2b234726"
server_key_iv: "37b14fe2c5202553325e298937921de4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:04.738209 1408 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260927 05:02:04.758795 1430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:04.760737 1408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 05:02:04.760984 1408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "e7e5da45ca934ea4b879d2a082a78b49"
format_stamp: "Formatted at 2026-09-27 05:02:04 on dist-test-slave-39f2"
server_key: "29abb764a8d1298c4dd02eec2b234726"
server_key_iv: "37b14fe2c5202553325e298937921de4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:04.761283 1408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:04.791782 1408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:04.797463 1408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:04.797852 1408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:04.799120 1408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:04.801137 1408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:04.801247 1408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:04.801384 1408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:04.801455 1408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:04.853776 1408 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:35093
I20260927 05:02:04.853778 1543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:35093 every 8 connection(s)
I20260927 05:02:04.855638 1408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:02:04.858343 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1408
I20260927 05:02:04.858546 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:02:04.859220 21157 external_mini_cluster.cc:1428] Setting key 03819d4e82fb03a667fa04c601096d0c
Sep 27 05:02:04 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485324, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:04.889106 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:04.858500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:42301 (local address 127.20.169.126:39321)
0927 05:02:04.869311 (+ 10811us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:04.869321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:04.870185 (+ 864us) server_negotiation.cc:407] Connection header received
0927 05:02:04.872604 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:04.872611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:04.872715 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:04.872889 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:04.874384 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.875573 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.877333 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.878160 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.878325 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:04.881754 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:04.881766 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:04.881774 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:04.881832 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:04.885857 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:04.886660 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:04.886665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:04.886667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:04.886792 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:04.887243 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:04.887247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:04.887249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:04.887518 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:04.887674 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:04.888430 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 05:02:04.888668 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10582,"thread_start_us":10448,"threads_started":1}
I20260927 05:02:04.890211 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:04.859361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39321 (local address 127.20.169.66:42301)
0927 05:02:04.869673 (+ 10312us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:04.869724 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:04.872195 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:04.873116 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:04.873126 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:04.873573 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:04.874184 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:04.874202 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.875737 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.875741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:04.877170 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:04.877181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:04.878449 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:04.878453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:04.878664 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:04.879397 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:04.879424 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:04.881495 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:04.886019 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:04.886026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:04.886040 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:04.886514 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:04.886903 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:04.886908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:04.886912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:04.887106 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:04.887646 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:04.887653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:04.887999 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 05:02:04.888323 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 05:02:04.888879 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10006,"thread_start_us":9732,"threads_started":1}
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
I20260927 05:02:04.891741 1544 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39321
I20260927 05:02:04.892058 1544 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:04.892993 1544 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
I20260927 05:02:04.894809 1185 ts_manager.cc:194] Registered new tserver with Master: e7e5da45ca934ea4b879d2a082a78b49 (127.20.169.66:35093)
I20260927 05:02:04.895694 1185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:42301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:02:04.924616 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39321
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:05.207327 1401 heartbeater.cc:503] Master 127.20.169.126:39321 was elected leader, sending a full tablet report...
W20260927 05:02:05.268035 1550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:05.268313 1550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:05.268494 1550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:05.275897 1550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:05.276012 1550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:05.276214 1550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:02:05.286123 1550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.169.84:43585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:05.287505 1550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:05.288911 1550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:05.302560 1555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:05.302524 1558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:05.302536 1556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:05.303270 1550 server_base.cc:1034] running on GCE node
I20260927 05:02:05.303802 1550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:05.304829 1550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:05.306087 1550 hybrid_clock.cc:648] HybridClock initialized: now 1790485325306030 us; error 57 us; skew 500 ppm
I20260927 05:02:05.306471 1550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:05 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485325, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:05.309988 1550 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:02:05.311511 1550 webserver.cc:460] Webserver started at http://127.20.169.67:33969/ using document root <none> and password file <none>
I20260927 05:02:05.312294 1550 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:05.312424 1550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:05.312789 1550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:02:05 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485325, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:05.330076 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "bb57d09f54ef4607b499bff04ae2440a"
format_stamp: "Formatted at 2026-09-27 05:02:05 on dist-test-slave-39f2"
server_key: "dda77bccd46e06fbd00c60e79d4fcacd"
server_key_iv: "e624fa5de1291c671b93e9555ebd0693"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:05.331140 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "bb57d09f54ef4607b499bff04ae2440a"
format_stamp: "Formatted at 2026-09-27 05:02:05 on dist-test-slave-39f2"
server_key: "dda77bccd46e06fbd00c60e79d4fcacd"
server_key_iv: "e624fa5de1291c671b93e9555ebd0693"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:05.338076 1550 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 05:02:05.355618 1565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:05.357529 1550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 05:02:05.357718 1550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "bb57d09f54ef4607b499bff04ae2440a"
format_stamp: "Formatted at 2026-09-27 05:02:05 on dist-test-slave-39f2"
server_key: "dda77bccd46e06fbd00c60e79d4fcacd"
server_key_iv: "e624fa5de1291c671b93e9555ebd0693"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:05.358000 1550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:05.382090 1550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:05.385998 1550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:05.386237 1550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:05.387096 1550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:05.388357 1550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:05.388432 1550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:05.388514 1550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:05.388557 1550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:05.422499 1550 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:32969
I20260927 05:02:05.422587 1677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:32969 every 8 connection(s)
I20260927 05:02:05.423987 1550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:02:05.430584 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1550
I20260927 05:02:05.430784 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:02:05.431293 21157 external_mini_cluster.cc:1428] Setting key f78d51e6fe442cd1fa264acdb765e0e7
Sep 27 05:02:05 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485325, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:05.445125 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:05.426598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56511 (local address 127.20.169.126:39321)
0927 05:02:05.427095 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:05.427100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:05.428024 (+ 924us) server_negotiation.cc:407] Connection header received
0927 05:02:05.429954 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:05.429961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:05.430103 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:05.430259 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:05.431888 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.432982 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.435131 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.435606 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.435711 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:05.439352 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:05.439357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:05.439360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:05.439393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:05.442196 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.442960 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.442965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.442967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.443085 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.443558 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.443561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.443563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.443820 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:05.443929 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:05.444567 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 05:02:05.444726 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260927 05:02:05.446221 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:05.427083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39321 (local address 127.20.169.67:56511)
0927 05:02:05.427822 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:05.427867 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:05.429658 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:05.430491 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:05.430500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:05.431037 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:05.431688 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:05.431708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.433137 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.433141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:05.434989 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:05.435002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.435848 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.435854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:05.436066 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:05.436731 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:05.436770 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:05.439098 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:05.442327 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:05.442334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:05.442347 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:05.442851 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:05.443200 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:05.443205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:05.443208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:05.443455 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:05.443930 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:05.443935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:05.444234 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 05:02:05.444472 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 05:02:05.444911 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":301,"threads_started":1}
I20260927 05:02:05.447685 1678 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39321
I20260927 05:02:05.448016 1678 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:05.448690 1678 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
I20260927 05:02:05.450249 1184 ts_manager.cc:194] Registered new tserver with Master: bb57d09f54ef4607b499bff04ae2440a (127.20.169.67:32969)
I20260927 05:02:05.451478 1184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:56511
I20260927 05:02:05.459717 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:02:05.461596 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1266
Sep 27 05:02:05 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:02:05.479646 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:05.462642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.20.169.65:45899)
0927 05:02:05.463394 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:05.463401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:05.463425 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:02:05.463481 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:05.463487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:05.463675 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:05.463795 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:05.465538 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.466628 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.468067 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.468671 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.468795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:05.473172 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:05.473183 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:05.473197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:05.473235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:05.476138 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.477038 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.477054 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.477057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.477229 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.477686 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.477698 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.477701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.478099 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:05.478485 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:05.478899 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 05:02:05.479224 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":391,"threads_started":1}
I20260927 05:02:05.481164 1395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:05.564612 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1266
I20260927 05:02:05.582170 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1408
Sep 27 05:02:05 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:02:05.598686 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:05.583022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.20.169.66:35093)
0927 05:02:05.583621 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:05.583628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:05.583654 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:02:05.583713 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:05.583720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:05.583939 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:05.584068 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:05.585466 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.586478 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.587812 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.588355 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.588451 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:05.593028 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:05.593038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:05.593052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:05.593082 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:05.596038 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.596678 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.596686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.596689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.596920 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.597355 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.597359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.597361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.597691 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:05.597943 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:05.598160 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 05:02:05.598403 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":344,"threads_started":1}
I20260927 05:02:05.599884 1539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:05.682808 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1408
I20260927 05:02:05.699709 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1550
Sep 27 05:02:05 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:02:05.717456 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:05.700897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.20.169.67:32969)
0927 05:02:05.701553 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:05.701564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:05.701606 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:02:05.701676 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:05.701683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:05.701928 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:05.702055 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:05.703831 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.705085 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.706701 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:05.707300 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:05.707422 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:05.711835 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:05.711845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:05.711910 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:05.711956 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:05.714916 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.715619 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.715625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.715627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.715744 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:05.716096 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:05.716099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:05.716101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:05.716411 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:05.716649 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:05.716986 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 05:02:05.717178 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":318,"threads_started":1}
I20260927 05:02:05.718830 1673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:05.803781 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1550
I20260927 05:02:05.820472 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1153
I20260927 05:02:05.821844 1215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:05.930833 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1153
I20260927 05:02:05.956553 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39321
--webserver_interface=127.20.169.126
--webserver_port=38601
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:39321
--ranger_config_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:06.301638 1690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:06.301947 1690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:06.302043 1690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:06.309362 1690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:06.309475 1690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:06.309535 1690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:06.309600 1690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:06.309648 1690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:06.320179 1690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:39321
--ranger_config_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=38601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:06.321455 1690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:06.322779 1690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:06.335222 1698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:06.335426 1695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:06.335347 1696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:06.336439 1690 server_base.cc:1034] running on GCE node
I20260927 05:02:06.337321 1690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:06.339401 1690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:06.340673 1690 hybrid_clock.cc:648] HybridClock initialized: now 1790485326340604 us; error 40 us; skew 500 ppm
I20260927 05:02:06.341202 1690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:06 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485326, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:06.345600 1690 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:06.347138 1690 webserver.cc:460] Webserver started at http://127.20.169.126:38601/ using document root <none> and password file <none>
I20260927 05:02:06.347787 1690 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:06.347905 1690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:02:06 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485326, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:06.411468 1690 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:02:06.414533 1706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:06.416042 1690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:02:06.416203 1690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "e677807975a840fca5c98330cfd8dc8e"
format_stamp: "Formatted at 2026-09-27 05:02:01 on dist-test-slave-39f2"
server_key: "3c748cead68b323a97f2116755866edb"
server_key_iv: "b2ccaa46a39b8b3734819235d0b4c8c4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:06.417202 1690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:06.431185 1690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:06.434957 1690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:06.435182 1690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:06.454108 1690 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:39321
I20260927 05:02:06.454131 1759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:39321 every 8 connection(s)
I20260927 05:02:06.455554 1690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:06.461126 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1690
I20260927 05:02:06.461997 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:45899
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:39321
--webserver_port=36597
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:06.462998 1760 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:02:06.466466 1760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Bootstrap starting.
I20260927 05:02:06.492676 1760 log.cc:824] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:06.505313 1760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:02:06.506009 1760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Bootstrap complete.
I20260927 05:02:06.520186 1760 raft_consensus.cc:348] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:06.521550 1760 raft_consensus.cc:729] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677807975a840fca5c98330cfd8dc8e, State: Initialized, Role: FOLLOWER
I20260927 05:02:06.522292 1760 consensus_queue.cc:260] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:06.522579 1760 raft_consensus.cc:388] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:06.522707 1760 raft_consensus.cc:482] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:06.522917 1760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:02:06.528578 1760 raft_consensus.cc:504] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:06.529312 1760 leader_election.cc:302] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e677807975a840fca5c98330cfd8dc8e; no voters:
I20260927 05:02:06.530109 1760 leader_election.cc:288] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:02:06.530340 1764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:02:06.531865 1764 raft_consensus.cc:686] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [term 2 LEADER]: Becoming Leader. State: Replica: e677807975a840fca5c98330cfd8dc8e, State: Running, Role: LEADER
I20260927 05:02:06.532464 1764 consensus_queue.cc:237] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } }
I20260927 05:02:06.532648 1760 sys_catalog.cc:564] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:06.535590 1766 sys_catalog.cc:455] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e677807975a840fca5c98330cfd8dc8e. Latest consensus state: current_term: 2 leader_uuid: "e677807975a840fca5c98330cfd8dc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } } }
I20260927 05:02:06.535921 1766 sys_catalog.cc:458] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:06.535250 1765 sys_catalog.cc:455] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e677807975a840fca5c98330cfd8dc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677807975a840fca5c98330cfd8dc8e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39321 } } }
I20260927 05:02:06.536473 1765 sys_catalog.cc:458] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:06.547027 1770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:06.553490 1770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:06.554971 1770 catalog_manager.cc:1259] Loaded cluster ID: 6d7c4590e4c14ac5a48fcd0acc827c5b
I20260927 05:02:06.555166 1770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:06.559417 1770 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:06.560984 1770 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Loaded TSK: 0
I20260927 05:02:06.562789 1770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:02:06.569802 1760 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 05:02:06.978835 1762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:06.979346 1762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:06.979579 1762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:06.992794 1762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:06.993021 1762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:06.993253 1762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:02:07.010924 1762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:45899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=36597
--tserver_master_addrs=127.20.169.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:07.012895 1762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:07.014876 1762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:07.033159 1797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:07.033470 1798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:07.035121 1800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:07.035810 1762 server_base.cc:1034] running on GCE node
I20260927 05:02:07.036412 1762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:07.037576 1762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:07.038806 1762 hybrid_clock.cc:648] HybridClock initialized: now 1790485327038738 us; error 51 us; skew 500 ppm
I20260927 05:02:07.039191 1762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:07.042791 1762 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:02:07.044416 1762 webserver.cc:460] Webserver started at http://127.20.169.65:36597/ using document root <none> and password file <none>
I20260927 05:02:07.045181 1762 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:07.045344 1762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:07.080140 1762 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.008s
I20260927 05:02:07.093428 1808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:07.096117 1762 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.010s
I20260927 05:02:07.096350 1762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "04bd557fd126400e96f11db2433334c5"
format_stamp: "Formatted at 2026-09-27 05:02:03 on dist-test-slave-39f2"
server_key: "2825e359fe60a3b15eef99b52fe548c6"
server_key_iv: "7ce53db2459edf2bec865352e4d01db9"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:07.097962 1762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:07.134107 1762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:07.141058 1762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:07.141425 1762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:07.142750 1762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:07.144972 1762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:07.145112 1762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:07.145339 1762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:07.145502 1762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:07.222038 1762 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:45899
I20260927 05:02:07.222081 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:45899 every 8 connection(s)
I20260927 05:02:07.223810 1762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:02:07.225986 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1762
I20260927 05:02:07.226962 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:35093
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:39321
--webserver_port=43291
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:07.256031 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:07.229140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:57725 (local address 127.20.169.126:39321)
0927 05:02:07.234608 (+ 5468us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:07.234632 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:07.234690 (+ 58us) server_negotiation.cc:407] Connection header received
0927 05:02:07.236136 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:07.236155 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:07.236666 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:07.237268 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:07.240049 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.241349 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.242886 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.243513 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.243714 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:07.247123 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:07.247130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:07.247145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:07.247180 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:07.250349 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:07.251101 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:07.251111 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:07.251120 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:07.251301 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:07.251849 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:07.251858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:07.251866 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:07.252637 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:07.252885 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:07.254149 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0927 05:02:07.254613 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5215,"thread_start_us":4992,"threads_started":1}
I20260927 05:02:07.256474 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:07.227809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39321 (local address 127.20.169.65:57725)
0927 05:02:07.230209 (+ 2400us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:07.230256 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:07.235334 (+ 5078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:07.237950 (+ 2616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:07.237964 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:07.238637 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:07.239502 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:07.239532 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.241508 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.241514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:07.242706 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:07.242725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.243745 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.243752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:07.244028 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:07.244739 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:07.244761 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:07.246879 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:07.250513 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:07.250523 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:07.250538 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:07.250950 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:07.251447 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:07.251452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:07.251456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:07.251693 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:07.253010 (+ 1317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:07.253024 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:07.253523 (+ 499us) client_negotiation.cc:769] Sending connection context
0927 05:02:07.253901 (+ 378us) client_negotiation.cc:240] Negotiation successful
0927 05:02:07.254582 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2077,"thread_start_us":1831,"threads_started":1}
I20260927 05:02:07.264200 1924 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39321
I20260927 05:02:07.264568 1924 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:07.265514 1924 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
W20260927 05:02:07.269050 1924 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:39321 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:02:07.269271 1924 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:07.269562 1924 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
I20260927 05:02:07.270731 1924 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:07.271162 1924 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
W20260927 05:02:07.272167 1924 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:02:07.638033 1928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:07.638305 1928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:07.638453 1928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:07.646061 1928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:07.646214 1928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:07.646418 1928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:02:07.657049 1928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:35093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=43291
--tserver_master_addrs=127.20.169.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:07.658383 1928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:07.659700 1928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:07.677548 1941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:07.678081 1940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:07.679574 1943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:07.680160 1928 server_base.cc:1034] running on GCE node
I20260927 05:02:07.680964 1928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:07.682284 1928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:07.683645 1928 hybrid_clock.cc:648] HybridClock initialized: now 1790485327683574 us; error 64 us; skew 500 ppm
I20260927 05:02:07.684067 1928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:07.687323 1928 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:02:07.688998 1928 webserver.cc:460] Webserver started at http://127.20.169.66:43291/ using document root <none> and password file <none>
I20260927 05:02:07.689762 1928 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:07.689929 1928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:07.722074 1928 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.002s sys 0.013s
I20260927 05:02:07.725426 1953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:07.727082 1928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:02:07.727273 1928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "e7e5da45ca934ea4b879d2a082a78b49"
format_stamp: "Formatted at 2026-09-27 05:02:04 on dist-test-slave-39f2"
server_key: "29abb764a8d1298c4dd02eec2b234726"
server_key_iv: "37b14fe2c5202553325e298937921de4"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:07.728739 1928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:07.757153 1928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:07.761410 1928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:07.761693 1928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:07.762804 1928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:07.764355 1928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:07.764480 1928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:07.764590 1928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:07.764695 1928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:07.908499 1928 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:35093
I20260927 05:02:07.908578 2068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:35093 every 8 connection(s)
I20260927 05:02:07.910693 1928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:02:07.914551 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 1928
I20260927 05:02:07.915431 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.169.84:43585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:32969
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:39321
--webserver_port=33969
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:02:07 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:07.966537 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:07.919773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:42781 (local address 127.20.169.126:39321)
0927 05:02:07.940363 (+ 20590us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:07.940370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:07.944428 (+ 4058us) server_negotiation.cc:407] Connection header received
0927 05:02:07.944501 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:07.944507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:07.944600 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:07.944706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:07.947483 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.948778 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.951555 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.952919 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.953021 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:07.958634 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:07.958642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:07.958647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:07.958690 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:07.962577 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:07.963606 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:07.963613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:07.963619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:07.963767 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:07.964329 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:07.964334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:07.964338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:07.964724 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:07.964876 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:07.965897 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0927 05:02:07.966131 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20417,"thread_start_us":21737,"threads_started":1}
I20260927 05:02:07.969409 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:07.920498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39321 (local address 127.20.169.66:42781)
0927 05:02:07.941054 (+ 20556us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:07.941116 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:07.943959 (+ 2843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:07.945402 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:07.945415 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:07.946367 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:07.947242 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:07.947272 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.949181 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.949187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:07.951380 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:07.951395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:07.953402 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:07.953409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:07.953699 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:07.954514 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:07.954545 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:07.958328 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:07.962790 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:07.962798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:07.962817 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:07.963459 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:07.963915 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:07.963920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:07.963924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:07.964189 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:07.964991 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:07.964999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:07.965458 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 05:02:07.966776 (+ 1318us) client_negotiation.cc:240] Negotiation successful
0927 05:02:07.967392 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20123,"thread_start_us":22462,"threads_started":1}
I20260927 05:02:07.977377 2069 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39321
I20260927 05:02:07.977806 2069 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:07.978832 2069 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
I20260927 05:02:07.981316 1724 ts_manager.cc:194] Registered new tserver with Master: e7e5da45ca934ea4b879d2a082a78b49 (127.20.169.66:35093)
I20260927 05:02:07.983600 1724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:42781
I20260927 05:02:08.272346 1924 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:08.272833 1924 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
I20260927 05:02:08.274245 1724 ts_manager.cc:194] Registered new tserver with Master: 04bd557fd126400e96f11db2433334c5 (127.20.169.65:45899)
I20260927 05:02:08.275455 1724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:57725
Sep 27 05:02:08 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
W20260927 05:02:08.422866 2073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:08.423233 2073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:08.423403 2073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:08.434023 2073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:08.434235 2073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:08.434481 2073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:02:08.450081 2073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.169.84:43585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:32969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=33969
--tserver_master_addrs=127.20.169.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:08.452168 2073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:08.453990 2073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:08.493093 2080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:08.497079 2073 server_base.cc:1034] running on GCE node
W20260927 05:02:08.497656 2079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:08.498118 2082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:08.501524 2073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:08.517153 2073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:08.518507 2073 hybrid_clock.cc:648] HybridClock initialized: now 1790485328518439 us; error 66 us; skew 500 ppm
I20260927 05:02:08.518973 2073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:08 dist-test-slave-39f2 krb5kdc[705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485328, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:08.526139 2073 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:02:08.528126 2073 webserver.cc:460] Webserver started at http://127.20.169.67:33969/ using document root <none> and password file <none>
I20260927 05:02:08.529062 2073 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:08.529400 2073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:02:08 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485328, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:08.552735 2073 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:02:08.556571 2090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:08.558547 2073 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:02:08.558722 2073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "bb57d09f54ef4607b499bff04ae2440a"
format_stamp: "Formatted at 2026-09-27 05:02:05 on dist-test-slave-39f2"
server_key: "dda77bccd46e06fbd00c60e79d4fcacd"
server_key_iv: "e624fa5de1291c671b93e9555ebd0693"
server_key_version: "kuduclusterkey@0"
I20260927 05:02:08.560143 2073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:08.576781 2073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:08.581845 2073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:08.582137 2073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:08.583329 2073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:08.585669 2073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:08.585774 2073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:08.585916 2073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:08.585994 2073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:08.630360 2073 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:32969
I20260927 05:02:08.630439 2202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:32969 every 8 connection(s)
I20260927 05:02:08.632191 2073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:02:08.640589 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2073
Sep 27 05:02:08 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485328, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:08.676360 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.635591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:57929 (local address 127.20.169.126:39321)
0927 05:02:08.649218 (+ 13627us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.649225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.649668 (+ 443us) server_negotiation.cc:407] Connection header received
0927 05:02:08.649752 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.649758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.649869 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.652199 (+ 2330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:08.653055 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.654622 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.657122 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.657838 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.657979 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.668447 (+ 10468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:08.668455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:08.668460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:08.668544 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:08.672066 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:08.673153 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:08.673158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:08.673162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:08.673296 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:08.673840 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:08.673845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:08.673848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:08.674205 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:08.674334 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.675723 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.675966 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13496,"thread_start_us":13379,"threads_started":1}
I20260927 05:02:08.677965 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.636253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39321 (local address 127.20.169.67:57929)
0927 05:02:08.637829 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:08.637897 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:08.641282 (+ 3385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:08.650896 (+ 9614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:08.650910 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:08.651530 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:08.652457 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.652484 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.654808 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.654814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.656931 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.656947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.658876 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.658884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.659196 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.660103 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:08.660137 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:08.665817 (+ 5680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:08.672255 (+ 6438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:08.672265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:08.672286 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:08.672997 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:08.673447 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:08.673453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:08.673456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:08.673703 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:08.674431 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:08.674439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:08.674887 (+ 448us) client_negotiation.cc:769] Sending connection context
0927 05:02:08.675257 (+ 370us) client_negotiation.cc:240] Negotiation successful
0927 05:02:08.675837 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"thread_start_us":939,"threads_started":1}
I20260927 05:02:08.679906 2203 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39321
I20260927 05:02:08.680234 2203 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:08.680907 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.649173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.20.169.126:39321)
0927 05:02:08.655169 (+ 5996us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.655173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.655187 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:02:08.655224 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.655227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.655302 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.655439 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:08.657627 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.658855 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.661005 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.661672 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.661795 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.665638 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:08.665645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:08.665650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:08.665690 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:08.676794 (+ 11104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:08.677807 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:08.677812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:08.677815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:08.677958 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:08.678444 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:08.678448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:08.678452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:08.678777 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:08.680132 (+ 1355us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.680267 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.680518 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5888,"thread_start_us":5777,"threads_started":1}
I20260927 05:02:08.681281 2203 heartbeater.cc:511] Master 127.20.169.126:39321 requested a full tablet report, sending...
I20260927 05:02:08.682968 1724 ts_manager.cc:194] Registered new tserver with Master: bb57d09f54ef4607b499bff04ae2440a (127.20.169.67:32969)
I20260927 05:02:08.684157 1724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:57929
I20260927 05:02:08.685027 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:02:08 dist-test-slave-39f2 krb5kdc[705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485327, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for HTTP/127.20.169.84@KRBTEST.COM
I20260927 05:02:08.716552 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.689565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.20.169.126:39321)
0927 05:02:08.689740 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.689746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.696188 (+ 6442us) server_negotiation.cc:407] Connection header received
0927 05:02:08.696646 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.696652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.696740 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.696845 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:08.699004 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.700122 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.702103 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.702757 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.702862 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.706437 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:08.706444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:08.706449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:08.706488 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:08.713163 (+ 6675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:08.714083 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:08.714089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:08.714093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:08.714248 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:08.714724 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:08.714729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:08.714732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:08.715094 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:08.715335 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.715953 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.716202 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:02:08.723220 1724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:02:08.734336 1724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:02:08.781873 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.751925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.20.169.66:35093)
0927 05:02:08.757731 (+ 5806us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.757738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.764585 (+ 6847us) server_negotiation.cc:407] Connection header received
0927 05:02:08.765354 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.765362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.765655 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.765870 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:08.766803 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.768410 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.771281 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.772725 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.772861 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.773077 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.781306 (+ 8229us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.781512 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5666,"thread_start_us":5528,"threads_started":1}
I20260927 05:02:08.783017 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.751258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:35093 (local address 127.0.0.1:55640)
0927 05:02:08.764366 (+ 13108us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:08.764411 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:08.764629 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:08.766166 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:08.766174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:08.766204 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:08.766614 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.766628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.768609 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.768614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.771104 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.771120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.773281 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.773287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.780979 (+ 7692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.781051 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 05:02:08.782430 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0927 05:02:08.782655 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12831,"thread_start_us":13607,"threads_started":1}
I20260927 05:02:08.799594 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.766030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.20.169.65:45899)
0927 05:02:08.773464 (+ 7434us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.773475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.790541 (+ 17066us) server_negotiation.cc:407] Connection header received
0927 05:02:08.790900 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.790910 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.791238 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.791463 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:08.792230 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.793944 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.796583 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.798122 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.798237 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.798500 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.799074 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.799249 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7261,"thread_start_us":7078,"threads_started":1}
I20260927 05:02:08.800559 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.769045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55854)
0927 05:02:08.790408 (+ 21363us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:08.790429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:08.790672 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:08.791805 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:08.791810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:08.791836 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:08.792038 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.792049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.794137 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.794143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.796094 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.796109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.798652 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.798658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.798847 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.798881 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:02:08.800074 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0927 05:02:08.800227 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21238,"thread_start_us":27291,"threads_started":1}
I20260927 05:02:08.808939 1858 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5464b00d1b4981892a4f2c0948006b (DEFAULT_TABLE table=test-table [id=d3ebb77708aa449e8a1c546ea496c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:08.809589 2003 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5464b00d1b4981892a4f2c0948006b (DEFAULT_TABLE table=test-table [id=d3ebb77708aa449e8a1c546ea496c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:08.810715 1858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5464b00d1b4981892a4f2c0948006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:08.810905 2003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5464b00d1b4981892a4f2c0948006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:08.827855 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.804416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:32969 (local address 127.0.0.1:33760)
0927 05:02:08.804584 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:08.804604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:08.804709 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:08.820429 (+ 15720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:08.820436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:08.820508 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:08.820712 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.820725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.823019 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.823024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.824928 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.824944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.827063 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.827068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.827258 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.827291 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:02:08.827367 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 05:02:08.827531 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 05:02:08.829115 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.797084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.20.169.67:32969)
0927 05:02:08.805026 (+ 7942us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.805036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.805073 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:02:08.805173 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.805181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.805490 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.812236 (+ 6746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:08.821001 (+ 8765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.822759 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.825110 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.826580 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.826681 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.826914 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.828233 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.828425 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7701,"thread_start_us":8649,"threads_started":1}
I20260927 05:02:08.833576 2138 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5464b00d1b4981892a4f2c0948006b (DEFAULT_TABLE table=test-table [id=d3ebb77708aa449e8a1c546ea496c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:08.835353 2138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5464b00d1b4981892a4f2c0948006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:08.860440 2226 tablet_bootstrap.cc:492] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Bootstrap starting.
I20260927 05:02:08.881137 2226 tablet_bootstrap.cc:654] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:08.882058 2227 tablet_bootstrap.cc:492] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Bootstrap starting.
I20260927 05:02:08.882113 2225 tablet_bootstrap.cc:492] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: Bootstrap starting.
I20260927 05:02:08.885493 2225 tablet_bootstrap.cc:654] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:08.886696 2225 log.cc:824] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:08.889169 2226 log.cc:824] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:08.893162 2225 tablet_bootstrap.cc:492] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: No bootstrap required, opened a new log
I20260927 05:02:08.893456 2226 tablet_bootstrap.cc:492] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: No bootstrap required, opened a new log
I20260927 05:02:08.893553 2225 ts_tablet_manager.cc:1397] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20260927 05:02:08.893815 2226 ts_tablet_manager.cc:1397] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.006s
I20260927 05:02:08.895409 2227 tablet_bootstrap.cc:654] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:08.896667 2227 log.cc:824] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:08.904253 2225 raft_consensus.cc:348] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.904753 2225 raft_consensus.cc:374] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:08.904924 2225 raft_consensus.cc:729] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bd557fd126400e96f11db2433334c5, State: Initialized, Role: FOLLOWER
I20260927 05:02:08.905735 2225 consensus_queue.cc:260] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.913062 2226 raft_consensus.cc:348] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.913626 2226 raft_consensus.cc:374] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:08.918799 1924 heartbeater.cc:503] Master 127.20.169.126:39321 was elected leader, sending a full tablet report...
I20260927 05:02:08.919109 2225 ts_tablet_manager.cc:1428] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: Time spent starting tablet: real 0.025s user 0.014s sys 0.010s
I20260927 05:02:08.926851 2227 tablet_bootstrap.cc:492] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: No bootstrap required, opened a new log
I20260927 05:02:08.927147 2227 ts_tablet_manager.cc:1397] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.005s
I20260927 05:02:08.932961 2226 raft_consensus.cc:729] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e5da45ca934ea4b879d2a082a78b49, State: Initialized, Role: FOLLOWER
I20260927 05:02:08.933815 2226 consensus_queue.cc:260] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.943913 2227 raft_consensus.cc:348] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.944465 2227 raft_consensus.cc:374] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:08.944576 2227 raft_consensus.cc:729] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb57d09f54ef4607b499bff04ae2440a, State: Initialized, Role: FOLLOWER
I20260927 05:02:08.945673 2069 heartbeater.cc:503] Master 127.20.169.126:39321 was elected leader, sending a full tablet report...
I20260927 05:02:08.945922 2227 consensus_queue.cc:260] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.946374 2226 ts_tablet_manager.cc:1428] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Time spent starting tablet: real 0.052s user 0.015s sys 0.004s
I20260927 05:02:08.957122 2203 heartbeater.cc:503] Master 127.20.169.126:39321 was elected leader, sending a full tablet report...
I20260927 05:02:08.961238 2227 ts_tablet_manager.cc:1428] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Time spent starting tablet: real 0.034s user 0.010s sys 0.014s
I20260927 05:02:08.976294 2231 raft_consensus.cc:482] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:08.976609 2231 raft_consensus.cc:504] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:08.978824 2231 leader_election.cc:288] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e5da45ca934ea4b879d2a082a78b49 (127.20.169.66:35093), bb57d09f54ef4607b499bff04ae2440a (127.20.169.67:32969)
W20260927 05:02:08.986577 1925 tablet.cc:2369] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:08.994374 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.979895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:39173 (local address 127.20.169.66:35093)
0927 05:02:08.980090 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.980096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.985907 (+ 5811us) server_negotiation.cc:407] Connection header received
0927 05:02:08.985991 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.985997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.986088 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.986184 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:08.987986 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.989131 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.991893 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.993132 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.993268 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.993342 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:08.993971 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 05:02:08.994109 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:02:08.995126 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.979405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:35093 (local address 127.20.169.65:39173)
0927 05:02:08.985335 (+ 5930us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:08.985358 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:08.985529 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:08.986677 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:08.986684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:08.986712 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:08.987125 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.987139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.989318 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.989324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.991703 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.991721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.993567 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.993574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.993770 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:08.993805 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:02:08.994656 (+ 851us) client_negotiation.cc:240] Negotiation successful
0927 05:02:08.994790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5780,"thread_start_us":5581,"threads_started":1}
I20260927 05:02:08.998787 2023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5464b00d1b4981892a4f2c0948006b" candidate_uuid: "04bd557fd126400e96f11db2433334c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e5da45ca934ea4b879d2a082a78b49" is_pre_election: true
I20260927 05:02:08.999380 2023 raft_consensus.cc:2454] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04bd557fd126400e96f11db2433334c5 in term 0.
I20260927 05:02:09.000334 1809 leader_election.cc:302] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04bd557fd126400e96f11db2433334c5, e7e5da45ca934ea4b879d2a082a78b49; no voters:
I20260927 05:02:09.000875 2231 raft_consensus.cc:2781] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:02:09.001015 2231 raft_consensus.cc:482] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:02:09.001097 2231 raft_consensus.cc:3037] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:09.002210 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.985653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:33375 (local address 127.20.169.67:32969)
0927 05:02:08.985850 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:08.985858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:08.995528 (+ 9670us) server_negotiation.cc:407] Connection header received
0927 05:02:08.995790 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:08.995796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:08.995876 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:08.995937 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:08.996480 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.997548 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.999818 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.001050 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.001190 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:09.001264 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:09.001831 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 05:02:09.001967 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:02:09.002835 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:08.986419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:32969 (local address 127.20.169.65:33375)
0927 05:02:08.995434 (+ 9015us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:08.995453 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:08.995627 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:08.996124 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:08.996129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:08.996154 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:08.996305 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.996315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:08.997743 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:08.997753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:08.999640 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:08.999655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.001423 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.001429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:09.001634 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:09.001668 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:02:09.002372 (+ 704us) client_negotiation.cc:240] Negotiation successful
0927 05:02:09.002546 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8906,"thread_start_us":11285,"threads_started":1}
I20260927 05:02:09.004315 2231 raft_consensus.cc:504] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:09.005015 2231 leader_election.cc:288] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [CANDIDATE]: Term 1 election: Requested vote from peers e7e5da45ca934ea4b879d2a082a78b49 (127.20.169.66:35093), bb57d09f54ef4607b499bff04ae2440a (127.20.169.67:32969)
I20260927 05:02:09.006141 2023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5464b00d1b4981892a4f2c0948006b" candidate_uuid: "04bd557fd126400e96f11db2433334c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e5da45ca934ea4b879d2a082a78b49"
I20260927 05:02:09.006376 2023 raft_consensus.cc:3037] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:09.009687 2023 raft_consensus.cc:2454] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04bd557fd126400e96f11db2433334c5 in term 1.
I20260927 05:02:09.009984 2158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5464b00d1b4981892a4f2c0948006b" candidate_uuid: "04bd557fd126400e96f11db2433334c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb57d09f54ef4607b499bff04ae2440a" is_pre_election: true
I20260927 05:02:09.010222 1809 leader_election.cc:302] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04bd557fd126400e96f11db2433334c5, e7e5da45ca934ea4b879d2a082a78b49; no voters:
I20260927 05:02:09.010497 2231 raft_consensus.cc:2781] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:09.010566 2158 raft_consensus.cc:2454] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04bd557fd126400e96f11db2433334c5 in term 0.
I20260927 05:02:09.009946 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5464b00d1b4981892a4f2c0948006b" candidate_uuid: "04bd557fd126400e96f11db2433334c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb57d09f54ef4607b499bff04ae2440a"
I20260927 05:02:09.011195 2157 raft_consensus.cc:3037] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:09.014701 2157 raft_consensus.cc:2454] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04bd557fd126400e96f11db2433334c5 in term 1.
I20260927 05:02:09.017120 2231 raft_consensus.cc:686] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 1 LEADER]: Becoming Leader. State: Replica: 04bd557fd126400e96f11db2433334c5, State: Running, Role: LEADER
I20260927 05:02:09.017795 2231 consensus_queue.cc:237] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } }
I20260927 05:02:09.031857 1724 catalog_manager.cc:5495] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04bd557fd126400e96f11db2433334c5 (127.20.169.65). New cstate: current_term: 1 leader_uuid: "04bd557fd126400e96f11db2433334c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04bd557fd126400e96f11db2433334c5" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 45899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 } health_report { overall_health: UNKNOWN } } }
I20260927 05:02:09.058136 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.051663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.20.169.65:45899)
0927 05:02:09.051874 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:09.051880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:09.052011 (+ 131us) server_negotiation.cc:407] Connection header received
0927 05:02:09.052270 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:09.052274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:09.052372 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:09.052554 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:02:09.053035 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.053756 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.055579 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.056224 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.056375 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:09.057008 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:02:09.057542 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:02:09.057728 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:09.057776 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 05:02:09.057856 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:02:09.067270 2023 raft_consensus.cc:1261] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 1 FOLLOWER]: Refusing update from remote peer 04bd557fd126400e96f11db2433334c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:09.067371 2157 raft_consensus.cc:1261] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 1 FOLLOWER]: Refusing update from remote peer 04bd557fd126400e96f11db2433334c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:09.068332 2236 consensus_queue.cc:1035] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb57d09f54ef4607b499bff04ae2440a" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:09.068732 2231 consensus_queue.cc:1035] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e5da45ca934ea4b879d2a082a78b49" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:09.083242 2247 mvcc.cc:205] Tried to move back new op lower bound from 7333827907849519104 to 7333827907661824000. Current Snapshot: MvccSnapshot[applied={T|T < 7333827907849519104}]
I20260927 05:02:09.086127 2248 mvcc.cc:205] Tried to move back new op lower bound from 7333827907849519104 to 7333827907661824000. Current Snapshot: MvccSnapshot[applied={T|T < 7333827907849519104}]
I20260927 05:02:09.104084 1724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:09.104375 1724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:09.108657 1724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e677807975a840fca5c98330cfd8dc8e: Sending DeleteTablet for 3 replicas of tablet 5e5464b00d1b4981892a4f2c0948006b
I20260927 05:02:09.110061 1858 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5464b00d1b4981892a4f2c0948006b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:09 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:55854
I20260927 05:02:09.110057 2138 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5464b00d1b4981892a4f2c0948006b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:09 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:33760
I20260927 05:02:09.110831 2003 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5464b00d1b4981892a4f2c0948006b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:09 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:55640
I20260927 05:02:09.120028 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1762
I20260927 05:02:09.146296 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.122595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.20.169.65:45899)
0927 05:02:09.122846 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:09.122854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:09.122871 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:02:09.122930 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:09.122935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:09.123011 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:09.123117 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:09.125299 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.126776 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.129215 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.130051 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.130240 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:09.135163 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:09.135186 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:09.135211 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:09.135268 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:09.140273 (+ 5005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:09.141443 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:09.141456 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:09.141462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:09.141686 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:09.142209 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:09.142217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:09.142223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:09.142788 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:09.143152 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:09.145458 (+ 2306us) server_negotiation.cc:299] Negotiation successful
0927 05:02:09.145781 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 05:02:09.146657 2258 tablet_replica.cc:330] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: stopping tablet replica
I20260927 05:02:09.147269 2258 raft_consensus.cc:2229] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:09.148003 1919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:09.148501 2256 tablet_replica.cc:330] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: stopping tablet replica
I20260927 05:02:09.148859 2258 raft_consensus.cc:2258] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:09.149848 2256 raft_consensus.cc:2229] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:09.153258 2256 raft_consensus.cc:2258] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:09.155773 2256 ts_tablet_manager.cc:1905] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:09.156021 2258 ts_tablet_manager.cc:1905] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:09.173270 2258 ts_tablet_manager.cc:1918] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:09.173592 2258 log.cc:1196] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/5e5464b00d1b4981892a4f2c0948006b
I20260927 05:02:09.174036 2256 ts_tablet_manager.cc:1918] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:09.174240 2258 ts_tablet_manager.cc:1939] T 5e5464b00d1b4981892a4f2c0948006b P e7e5da45ca934ea4b879d2a082a78b49: Deleting consensus metadata
I20260927 05:02:09.174289 2256 log.cc:1196] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/5e5464b00d1b4981892a4f2c0948006b
I20260927 05:02:09.174834 2256 ts_tablet_manager.cc:1939] T 5e5464b00d1b4981892a4f2c0948006b P bb57d09f54ef4607b499bff04ae2440a: Deleting consensus metadata
I20260927 05:02:09.176324 1707 catalog_manager.cc:4841] TS e7e5da45ca934ea4b879d2a082a78b49 (127.20.169.66:35093): tablet 5e5464b00d1b4981892a4f2c0948006b (table test-table [id=d3ebb77708aa449e8a1c546ea496c6cc]) successfully deleted
I20260927 05:02:09.176704 1707 catalog_manager.cc:4841] TS bb57d09f54ef4607b499bff04ae2440a (127.20.169.67:32969): tablet 5e5464b00d1b4981892a4f2c0948006b (table test-table [id=d3ebb77708aa449e8a1c546ea496c6cc]) successfully deleted
I20260927 05:02:09.294411 2257 tablet_replica.cc:330] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5: stopping tablet replica
I20260927 05:02:09.294927 2257 raft_consensus.cc:2229] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:02:09.295353 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1762
I20260927 05:02:09.295670 2257 raft_consensus.cc:2258] T 5e5464b00d1b4981892a4f2c0948006b P 04bd557fd126400e96f11db2433334c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:09.314322 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1928
W20260927 05:02:09.314244 1707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:45899 (error 108)
I20260927 05:02:09.316207 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.315727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55884)
0927 05:02:09.315925 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.316049 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 05:02:09.316641 1707 catalog_manager.cc:4584] TS 04bd557fd126400e96f11db2433334c5 (127.20.169.65:45899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e5464b00d1b4981892a4f2c0948006b: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
I20260927 05:02:09.328536 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.315390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.20.169.66:35093)
0927 05:02:09.315600 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:09.315608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:09.315630 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:02:09.315701 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:09.315705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:09.315798 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:09.315903 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:09.317595 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.318316 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.319596 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.319972 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.320134 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:09.322688 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:09.322699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:09.322714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:09.322749 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:09.325753 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:09.326446 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:09.326453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:09.326455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:09.326587 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:09.326996 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:09.327003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:09.327011 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:09.327342 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:09.327648 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:09.327942 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 05:02:09.328209 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:02:09.329306 2064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:09.370728 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.370150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55892)
0927 05:02:09.370394 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.370503 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 05:02:09.371995 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.371517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55908)
0927 05:02:09.371733 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.371849 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 05:02:09.418898 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1928
I20260927 05:02:09.433058 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.432476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55924)
0927 05:02:09.432717 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.432802 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 05:02:09.434044 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.433698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55928)
0927 05:02:09.433845 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.433935 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 05:02:09.437933 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2073
I20260927 05:02:09.452714 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.438643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.20.169.67:32969)
0927 05:02:09.438810 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:09.438815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:09.438833 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:02:09.438914 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:09.438917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:09.438991 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:09.439091 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:09.440648 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.441495 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.443048 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:09.443522 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:09.443635 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:09.446281 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:09.446297 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:09.446323 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:09.446376 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:09.449585 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:09.450361 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:09.450369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:09.450372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:09.450530 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:09.450922 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:09.450926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:09.450928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:09.451285 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:09.451740 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:09.452096 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 05:02:09.452358 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:02:09.453470 2198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:09.515012 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.514486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55936)
0927 05:02:09.514716 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.514822 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 05:02:09.516227 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:09.515782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:45899 (local address 127.0.0.1:55940)
0927 05:02:09.516029 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:09.516100 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 05:02:09.557016 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2073
I20260927 05:02:09.574388 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 1690
I20260927 05:02:09.575713 1755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:09.688737 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 1690
I20260927 05:02:09.714112 21157 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 05:02:10.059710 21157 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 05:02:10.059983 21157 mini_ranger.cc:66] Stopping Ranger...
I20260927 05:02:10.405088 21157 mini_ranger.cc:68] Stopped Ranger
2026-09-27 05:02:10.405 UTC [735] LOG: received smart shutdown request
2026-09-27 05:02:10.408 UTC [735] LOG: background worker "logical replication launcher" (PID 744) exited with exit code 1
2026-09-27 05:02:10.408 UTC [739] LOG: shutting down
2026-09-27 05:02:10.590 UTC [735] LOG: database system is shut down
2026-09-27T05:02:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53671 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:02:10 dist-test-slave-39f2 krb5kdc[2272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:02:10 dist-test-slave-39f2 krb5kdc[2272](info): set up 2 sockets
Sep 27 05:02:10 dist-test-slave-39f2 krb5kdc[2272](info): commencing operation
krb5kdc: starting...
W20260927 05:02:11.840919 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:02:12 dist-test-slave-39f2 krb5kdc[2272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:02:12.085991 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:37069
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:37069,127.20.169.125:44923,127.20.169.124:38463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:12.414075 2285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:12.414362 2285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:12.414541 2285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:12.421687 2285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:12.421777 2285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:12.421854 2285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:12.421962 2285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:12.422026 2285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:12.431496 2285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:37069,127.20.169.125:44923,127.20.169.124:38463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:37069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:12.432781 2285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:12.434073 2285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:12.446173 2291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:12.446197 2293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:12.446794 2290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:12.447427 2285 server_base.cc:1034] running on GCE node
I20260927 05:02:12.448237 2285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:12.449834 2285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:12.451092 2285 hybrid_clock.cc:648] HybridClock initialized: now 1790485332451041 us; error 55 us; skew 500 ppm
I20260927 05:02:12.451517 2285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:12 dist-test-slave-39f2 krb5kdc[2272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:12.454743 2285 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:12.456465 2285 webserver.cc:460] Webserver started at http://127.20.169.126:46483/ using document root <none> and password file <none>
I20260927 05:02:12.457319 2285 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:12.457476 2285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:12.457816 2285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:12.461196 2285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "deb02cd22c6d426db761470825933ced"
format_stamp: "Formatted at 2026-09-27 05:02:12 on dist-test-slave-39f2"
server_key: "fc9003e91e7489a3e7dba412068119e0"
server_key_iv: "4a1d771366514e9c58a6af3cf0cd642b"
server_key_version: "encryptionkey@0"
I20260927 05:02:12.462090 2285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "deb02cd22c6d426db761470825933ced"
format_stamp: "Formatted at 2026-09-27 05:02:12 on dist-test-slave-39f2"
server_key: "fc9003e91e7489a3e7dba412068119e0"
server_key_iv: "4a1d771366514e9c58a6af3cf0cd642b"
server_key_version: "encryptionkey@0"
I20260927 05:02:12.467588 2285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:02:12.471391 2300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:12.472802 2285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:02:12.473062 2285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "deb02cd22c6d426db761470825933ced"
format_stamp: "Formatted at 2026-09-27 05:02:12 on dist-test-slave-39f2"
server_key: "fc9003e91e7489a3e7dba412068119e0"
server_key_iv: "4a1d771366514e9c58a6af3cf0cd642b"
server_key_version: "encryptionkey@0"
I20260927 05:02:12.473258 2285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:12.493911 2285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:12.498611 2285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:12.498935 2285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:12.519328 2285 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:37069
I20260927 05:02:12.519354 2351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:37069 every 8 connection(s)
I20260927 05:02:12.520859 2285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:12.524921 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:12.529238 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2285
I20260927 05:02:12.529498 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:02:12.530232 21157 external_mini_cluster.cc:1428] Setting key d6ba29c3345ea389cdf18e382cab33ca
I20260927 05:02:12.532135 2352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:02:12 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.125@KRBTEST.COM" created.
I20260927 05:02:12.560032 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.535433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:37069 (local address 127.0.0.1:47606)
0927 05:02:12.536335 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.536391 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:12.538736 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:12.539786 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:12.539793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:12.540397 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:12.540747 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:12.540774 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:12.543520 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:12.543525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:12.544754 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:12.544779 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:12.546402 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:12.546408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:12.546550 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:12.547274 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:12.547297 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:12.550330 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:12.555078 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:12.555089 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:12.555099 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:12.555572 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:12.556092 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:12.556096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:12.556098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:12.556272 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:12.557188 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:12.557198 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:12.557489 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 05:02:12.557796 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 05:02:12.558455 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":178,"thread_start_us":349,"threads_started":1}
I20260927 05:02:12.560053 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.536583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.20.169.126:37069)
0927 05:02:12.537336 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:12.537342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:12.537384 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:02:12.539134 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:12.539145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:12.539486 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:12.539725 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:12.541017 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:12.543337 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:12.544972 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:12.546113 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:12.546343 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:12.550647 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:12.550661 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:12.550683 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:12.550728 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:12.554923 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:12.555740 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:12.555747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:12.555749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:12.555971 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:12.556396 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:12.556400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:12.556402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:12.557014 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:12.557206 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:12.557934 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 05:02:12.558345 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":430,"threads_started":1}
I20260927 05:02:12.563694 2352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:02:12.565029 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.564541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58836)
0927 05:02:12.564756 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.564922 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 05:02:12.565354 2302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
I20260927 05:02:12.566932 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.566592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58840)
0927 05:02:12.566732 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.566808 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:02:12.567143 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:44923: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab.
Entry for principal kudu/127.20.169.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab.
I20260927 05:02:12.585959 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:44923
--webserver_interface=127.20.169.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab
--principal=kudu/127.20.169.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:37069,127.20.169.125:44923,127.20.169.124:38463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:12.616454 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } attempt: 1
I20260927 05:02:12.617734 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.617229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58848)
0927 05:02:12.617475 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.617601 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 05:02:12.618854 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.618446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58862)
0927 05:02:12.618609 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.618689 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:02:12.619055 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:44923: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
I20260927 05:02:12.687355 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } attempt: 2
I20260927 05:02:12.688618 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.688086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58874)
0927 05:02:12.688365 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.688468 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 05:02:12.689766 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.689366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58882)
0927 05:02:12.689553 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.689623 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:02:12.689965 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:44923: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
I20260927 05:02:12.781240 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } attempt: 3
I20260927 05:02:12.782454 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.781955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58898)
0927 05:02:12.782168 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.782268 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 05:02:12.783514 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.783097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58912)
0927 05:02:12.783244 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.783324 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:02:12.783684 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:44923: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
W20260927 05:02:12.911682 2360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:12.912065 2360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:12.912231 2360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:12.919469 2360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:12.919613 2360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:12.919727 2360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:12.919831 2360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:12.919916 2360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:12.926911 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } attempt: 4
I20260927 05:02:12.928210 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.927575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58918)
0927 05:02:12.927932 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.928042 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 05:02:12.929363 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:12.928945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58934)
0927 05:02:12.929107 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:12.929213 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 05:02:12.929580 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:44923: Network error: Client connection negotiation failed: client connection to 127.20.169.125:44923: connect: Connection refused (error 111)
I20260927 05:02:12.930703 2360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:37069,127.20.169.125:44923,127.20.169.124:38463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab
--principal=kudu/127.20.169.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.169.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:12.932052 2360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:12.933506 2360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:12.945430 2368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:12.945448 2366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:12.945389 2365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:12.946368 2360 server_base.cc:1034] running on GCE node
I20260927 05:02:12.947079 2360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:12.948453 2360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:12.949689 2360 hybrid_clock.cc:648] HybridClock initialized: now 1790485332949653 us; error 59 us; skew 500 ppm
I20260927 05:02:12.950052 2360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:12 dist-test-slave-39f2 krb5kdc[2272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:12.953338 2360 init.cc:377] Logged in from keytab as kudu/127.20.169.125@KRBTEST.COM (short username kudu)
I20260927 05:02:12.954860 2360 webserver.cc:460] Webserver started at http://127.20.169.125:33431/ using document root <none> and password file <none>
I20260927 05:02:12.955559 2360 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:12.955683 2360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:12.955999 2360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:12.959025 2360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data/instance:
uuid: "89fe910ccd35475db59a07bc90337052"
format_stamp: "Formatted at 2026-09-27 05:02:12 on dist-test-slave-39f2"
server_key: "f55e8269c442c89b55911bfe0e24748c"
server_key_iv: "a465a34288617cd7c81351de639ee135"
server_key_version: "encryptionkey@0"
I20260927 05:02:12.959990 2360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal/instance:
uuid: "89fe910ccd35475db59a07bc90337052"
format_stamp: "Formatted at 2026-09-27 05:02:12 on dist-test-slave-39f2"
server_key: "f55e8269c442c89b55911bfe0e24748c"
server_key_iv: "a465a34288617cd7c81351de639ee135"
server_key_version: "encryptionkey@0"
I20260927 05:02:12.965155 2360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 05:02:12.968557 2375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:12.970129 2360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:12.970330 2360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal
uuid: "89fe910ccd35475db59a07bc90337052"
format_stamp: "Formatted at 2026-09-27 05:02:12 on dist-test-slave-39f2"
server_key: "f55e8269c442c89b55911bfe0e24748c"
server_key_iv: "a465a34288617cd7c81351de639ee135"
server_key_version: "encryptionkey@0"
I20260927 05:02:12.970530 2360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:12.992993 2360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:12.997803 2360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:12.998121 2360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:13.017658 2360 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.125:44923
I20260927 05:02:13.017678 2426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.125:44923 every 8 connection(s)
I20260927 05:02:13.019140 2360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
I20260927 05:02:13.022814 2427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:13.027974 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2360
I20260927 05:02:13.028187 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-1/wal/instance
I20260927 05:02:13.028658 21157 external_mini_cluster.cc:1428] Setting key df74a843ee68e2b17fbb31d4240e5ea6
I20260927 05:02:13.029259 2427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.124@KRBTEST.COM; defaulting to no policy
I20260927 05:02:13.053225 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.031891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.20.169.126:37069)
0927 05:02:13.032078 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.032083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.033505 (+ 1422us) server_negotiation.cc:407] Connection header received
0927 05:02:13.035540 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.035546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.035643 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.035767 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.038107 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.038844 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.041656 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.042127 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.042269 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.046838 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.046850 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.046853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.046895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.050031 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.050877 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.050888 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.050891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.051010 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.051449 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.051458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.051460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.051743 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.051908 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.052636 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.052897 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:02:13.054180 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.032426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:37069 (local address 127.0.0.1:47618)
0927 05:02:13.033334 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.033381 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.035238 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.036023 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.036032 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.036942 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.037872 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.037901 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.039023 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.039033 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.041467 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.041486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.042390 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.042400 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.042728 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.043578 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.043615 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.046554 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.050203 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.050212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.050231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.050730 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.051125 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.051130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.051134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.051317 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.051880 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.051889 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.052234 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.052538 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.053095 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":395,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.124@KRBTEST.COM" created.
I20260927 05:02:13.056059 2427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:02:13.072445 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.056891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58942)
0927 05:02:13.057138 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.057161 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.057274 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.058240 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.058248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.058757 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.058993 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.059008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.060490 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.060498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.061966 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.061994 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.063201 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.063206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.063446 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.064123 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.064162 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.066778 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.070011 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.070018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.070022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.070379 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.070761 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.070767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.070770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.070905 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.071492 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.071497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.071638 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.071783 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.072052 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 05:02:13.072567 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.057112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.20.169.125:44923)
0927 05:02:13.057815 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.057823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.057848 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:02:13.057891 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.057896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.058067 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.058181 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.059154 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.060352 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.062138 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.063044 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.063207 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.066994 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.067005 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.067018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.067049 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.069847 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.070515 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.070531 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.070533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.070654 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.071027 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.071030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.071032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.071355 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.071576 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.071913 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.072164 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":532,"thread_start_us":393,"threads_started":1}
I20260927 05:02:13.075572 2427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:02:13.076659 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.076224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40662)
0927 05:02:13.076376 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.076521 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:02:13.076921 2377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.078490 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.078160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40674)
0927 05:02:13.078325 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.078397 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:02:13.078694 2427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab.
Entry for principal kudu/127.20.169.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab.
I20260927 05:02:13.088199 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:38463
--webserver_interface=127.20.169.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab
--principal=kudu/127.20.169.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:37069,127.20.169.125:44923,127.20.169.124:38463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:13.127993 2427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 1
I20260927 05:02:13.129274 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.128661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40676)
0927 05:02:13.128958 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.129085 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:02:13.130357 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.129949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40686)
0927 05:02:13.130135 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.130224 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 05:02:13.130631 2427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.198854 2427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 2
I20260927 05:02:13.200086 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.199644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40694)
0927 05:02:13.199837 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.199926 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 05:02:13.201222 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.200914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40698)
0927 05:02:13.201048 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.201118 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 05:02:13.201514 2427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.228933 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } attempt: 5
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:02:13.243038 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.229880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.20.169.125:44923)
0927 05:02:13.230079 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.230084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.230104 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:02:13.230316 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.230320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.230383 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.230467 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.231547 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.232213 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.233897 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.234330 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.234426 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.237906 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.237910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.237914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.237944 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.240395 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.241143 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.241151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.241154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.241285 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.241693 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.241698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.241701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.241974 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.242131 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.242520 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.242731 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:02:13.243038 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.229649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58946)
0927 05:02:13.229965 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.229994 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.230169 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.230578 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.230585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.231113 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.231374 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.231387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.232355 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.232361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.233749 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.233766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.234477 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.234483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.234676 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.235301 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.235324 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.237716 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.240557 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.240564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.240568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.241026 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.241398 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.241403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.241407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.241571 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.242097 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.242102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.242252 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.242389 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.242654 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 05:02:13.244063 2352 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:02:13.245016 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.244631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40706)
0927 05:02:13.244789 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.244895 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:02:13.245936 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.245622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40718)
0927 05:02:13.245775 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.245846 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:02:13.246163 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.292737 2427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 3
I20260927 05:02:13.294080 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.293581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40726)
0927 05:02:13.293805 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.293898 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 05:02:13.295267 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.294844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40736)
0927 05:02:13.295075 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.295156 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260927 05:02:13.295512 2427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.297401 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 1
I20260927 05:02:13.298449 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.297957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40738)
0927 05:02:13.298157 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.298273 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:02:13.299491 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.299081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40750)
0927 05:02:13.299248 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.299336 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:02:13.299695 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.367913 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 2
I20260927 05:02:13.369163 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.368569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40760)
0927 05:02:13.368842 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.368958 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:02:13.370429 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.369974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40774)
0927 05:02:13.370162 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.370271 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 05:02:13.370661 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
W20260927 05:02:13.419413 2434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:13.419697 2434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:13.419870 2434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:13.426998 2434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:13.427090 2434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:13.427186 2434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:13.427310 2434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:13.427379 2434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:13.437021 2434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:37069,127.20.169.125:44923,127.20.169.124:38463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab
--principal=kudu/127.20.169.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:38463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.169.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:13.438302 2434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:13.438772 2427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 4
I20260927 05:02:13.439931 2434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 05:02:13.440125 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.439587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40784)
0927 05:02:13.439824 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.439911 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 05:02:13.441295 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.440974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40786)
0927 05:02:13.441120 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.441196 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:02:13.441526 2427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
W20260927 05:02:13.451879 2442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:13.451997 2440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:13.452023 2439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:13.453063 2434 server_base.cc:1034] running on GCE node
I20260927 05:02:13.453802 2434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:13.455065 2434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:13.456308 2434 hybrid_clock.cc:648] HybridClock initialized: now 1790485333456208 us; error 42 us; skew 500 ppm
I20260927 05:02:13.456656 2434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485333, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:13.459753 2434 init.cc:377] Logged in from keytab as kudu/127.20.169.124@KRBTEST.COM (short username kudu)
I20260927 05:02:13.461187 2434 webserver.cc:460] Webserver started at http://127.20.169.124:35211/ using document root <none> and password file <none>
I20260927 05:02:13.461838 2434 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:13.461935 2434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:13.462167 2434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:13.465018 2434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data/instance:
uuid: "c454355fe74043c0aaca404263f2394c"
format_stamp: "Formatted at 2026-09-27 05:02:13 on dist-test-slave-39f2"
server_key: "76d1384b3f56ff568493c628237409b0"
server_key_iv: "a0b70e79f838bad291f62b4ad047fa83"
server_key_version: "encryptionkey@0"
I20260927 05:02:13.465849 2434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal/instance:
uuid: "c454355fe74043c0aaca404263f2394c"
format_stamp: "Formatted at 2026-09-27 05:02:13 on dist-test-slave-39f2"
server_key: "76d1384b3f56ff568493c628237409b0"
server_key_iv: "a0b70e79f838bad291f62b4ad047fa83"
server_key_version: "encryptionkey@0"
I20260927 05:02:13.471019 2434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:02:13.474436 2449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:13.475859 2434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:02:13.476027 2434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal
uuid: "c454355fe74043c0aaca404263f2394c"
format_stamp: "Formatted at 2026-09-27 05:02:13 on dist-test-slave-39f2"
server_key: "76d1384b3f56ff568493c628237409b0"
server_key_iv: "a0b70e79f838bad291f62b4ad047fa83"
server_key_version: "encryptionkey@0"
I20260927 05:02:13.476282 2434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:13.476922 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 3
I20260927 05:02:13.478134 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.477645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40794)
0927 05:02:13.477874 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.477975 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:02:13.479103 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.478752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40806)
0927 05:02:13.478915 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.478998 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:02:13.479332 2352 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:38463: Network error: Client connection negotiation failed: client connection to 127.20.169.124:38463: connect: Connection refused (error 111)
I20260927 05:02:13.497102 2434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:13.501044 2434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:13.501310 2434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:13.520231 2434 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.124:38463
I20260927 05:02:13.520323 2500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.124:38463 every 8 connection(s)
I20260927 05:02:13.521621 2434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
I20260927 05:02:13.526106 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:13.530416 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2434
I20260927 05:02:13.530606 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485126081201-21157-0/minicluster-data/master-2/wal/instance
I20260927 05:02:13.531109 21157 external_mini_cluster.cc:1428] Setting key 5cfb1261157cd57caeb9ec02095e239a
I20260927 05:02:13.533416 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485333, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:02:13.554942 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.537236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.20.169.126:37069)
0927 05:02:13.537450 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.537457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.538121 (+ 664us) server_negotiation.cc:407] Connection header received
0927 05:02:13.538333 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.538339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.538424 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.538520 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.539680 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.540533 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.542169 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.542610 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.542745 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.548521 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.548526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.548529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.548562 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.551487 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.552490 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.552494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.552496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.552636 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.553121 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.553130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.553132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.553404 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.553577 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.554358 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.554585 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:02:13.555936 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.533043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.20.169.124:38463)
0927 05:02:13.533802 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.533814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.533855 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:02:13.535446 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.535467 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.535885 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.536254 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.537976 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.539509 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.541130 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.541853 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.541988 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.549316 (+ 7328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.549324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.549337 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.549367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.552419 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.553208 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.553215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.553217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.553333 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.553709 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.553713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.553715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.554111 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.554409 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.554737 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.554991 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":556,"spinlock_wait_cycles":19328,"thread_start_us":344,"threads_started":1}
I20260927 05:02:13.555981 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.537209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:37069 (local address 127.0.0.1:47628)
0927 05:02:13.537965 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.538012 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.538173 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.538583 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.538588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.539161 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.539493 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.539503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.540700 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.540707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.542005 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.542018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.542860 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.542868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.543007 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.545990 (+ 2983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.546017 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.548274 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.551619 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.551631 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.551643 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.552366 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.552764 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.552773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.552777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.552994 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.553546 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.553556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.553923 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.554208 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.554684 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":3,"thread_start_us":378,"threads_started":1}
I20260927 05:02:13.557412 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485333, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:13.571424 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.558255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:44923 (local address 127.0.0.1:58952)
0927 05:02:13.558466 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.558481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.558565 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.559334 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.559340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.559689 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.559828 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.559836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.560764 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.560772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.561851 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.561864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.562626 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.562631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.562844 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.563417 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.563434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.565809 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.569015 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.569021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.569025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.569393 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.569766 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.569771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.569774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.569917 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.570444 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.570449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.570611 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.570765 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.571066 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 05:02:13.571456 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.558156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.20.169.125:44923)
0927 05:02:13.558364 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.558369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.558554 (+ 185us) server_negotiation.cc:407] Connection header received
0927 05:02:13.559097 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.559105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.559186 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.559295 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.559963 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.560635 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.562009 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.562489 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.562619 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.566007 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.566018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.566021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.566052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.568881 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.569518 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.569523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.569527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.569654 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.570037 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.570043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.570046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.570330 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.570482 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.570868 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.571073 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:02:13.572914 2501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:02:13.573565 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.559036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.20.169.126:37069)
0927 05:02:13.559215 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.559221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.559238 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:02:13.559320 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.559324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.559398 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.559500 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.560681 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.561546 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.562820 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.563232 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.563370 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.567667 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.567672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.567675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.567707 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.570428 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.571177 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.571183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.571186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.571325 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.571764 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.571768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.571770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.572146 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.572384 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.572924 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.573147 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485333, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:02:13.587451 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.573949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.20.169.124:38463)
0927 05:02:13.574121 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.574127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.574168 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:02:13.574570 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.574575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.574634 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.574783 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.575478 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.576391 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.578026 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.578460 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.578604 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.581887 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.581894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.581898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.581927 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.584757 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.585441 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.585450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.585455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.585582 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.585965 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.585973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.585977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.586260 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.586425 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.586770 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.586996 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:02:13.587574 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.573715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40818)
0927 05:02:13.574074 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.574113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.574239 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.574857 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.574862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.575188 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.575322 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.575333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.576539 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.576547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.577816 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.577829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.578620 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.578626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.578846 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.579458 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.579477 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.581701 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.584945 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.584952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.584956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.585313 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.585710 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.585721 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.585723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.585856 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.586382 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.586387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.586543 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.586691 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.587158 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":148}
I20260927 05:02:13.590503 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.575646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.20.169.125:44923)
0927 05:02:13.575792 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.575796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.576024 (+ 228us) server_negotiation.cc:407] Connection header received
0927 05:02:13.576202 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.576207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.576269 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.576374 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.577651 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.578367 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.579563 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.580072 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.580197 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.584962 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.584975 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.584979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.585021 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.587650 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.588469 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.588474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.588477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.588667 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.589183 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.589189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.589192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.589488 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.589697 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.590069 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.590228 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:02:13.593883 2501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: Bootstrap starting.
I20260927 05:02:13.595551 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:02:13.598179 2501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:13.599640 2501 log.cc:824] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:13.603524 2501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: No bootstrap required, opened a new log
I20260927 05:02:13.615450 2501 raft_consensus.cc:348] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:13.615947 2501 raft_consensus.cc:374] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:13.616109 2501 raft_consensus.cc:729] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c454355fe74043c0aaca404263f2394c, State: Initialized, Role: FOLLOWER
I20260927 05:02:13.618086 2501 consensus_queue.cc:260] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:13.622922 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.602352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.20.169.126:37069)
0927 05:02:13.602542 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.602547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.602669 (+ 122us) server_negotiation.cc:407] Connection header received
0927 05:02:13.602995 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.603001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.603088 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.603170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.605913 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.607074 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.609555 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.610297 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.610421 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.615025 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.615032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.615036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.615082 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.619145 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.620354 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.620360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.620363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.620549 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.621036 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.621043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.621047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.621419 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.621663 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.622262 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.622525 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:02:13.623075 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:13.623591 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:13.624646 2501 sys_catalog.cc:564] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:13.626597 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.601782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.20.169.125:44923)
0927 05:02:13.602175 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.602180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.603007 (+ 827us) server_negotiation.cc:407] Connection header received
0927 05:02:13.603301 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.603306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.603387 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.603464 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.606434 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.607907 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.610813 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.611701 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.611832 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.617552 (+ 5720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.617560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.617564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.617606 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.622094 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.624015 (+ 1921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.624021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.624024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.624213 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.624780 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.624785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.624788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.625161 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.625436 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.625934 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.626182 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260927 05:02:13.633790 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.603372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.20.169.124:38463)
0927 05:02:13.603529 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.603536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.614708 (+ 11172us) server_negotiation.cc:407] Connection header received
0927 05:02:13.615037 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.615044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.615129 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.615204 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.617381 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.618514 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.620514 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.621217 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.621323 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.625557 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.625564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.625570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.625619 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.630210 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.630957 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.630967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.630973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.631209 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.631621 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.631630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.631636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.632178 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.632654 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.632883 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.633181 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 05:02:13.639459 2525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:02:13.639644 2525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:02:13.636461 2434 master_runner.cc:186] Error getting master registration for 127.20.169.126:37069: OK, instance_id { permanent_uuid: "deb02cd22c6d426db761470825933ced" instance_seqno: 1790485332513461 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:02:13.644393 2434 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:02:13.656564 2352 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 4
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:02:13.671747 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.657254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.20.169.124:38463)
0927 05:02:13.657457 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.657461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.657480 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:02:13.657667 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.657670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.657767 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.657892 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.658910 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.660204 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.661875 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.662326 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.662471 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.665747 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.665752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.665755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.665788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.668935 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.669741 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.669747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.669751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.669928 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.670434 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.670438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.670441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.670732 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.670856 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.671321 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.671499 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:02:13.671923 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.657186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40832)
0927 05:02:13.657403 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.657424 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.657535 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.657923 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.657929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.658471 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.658716 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.658728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.660376 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.660384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.661725 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.661737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.662472 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.662478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.662682 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.663334 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.663351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.665567 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.669081 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.669086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.669089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.669610 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.670102 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.670109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.670113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.670285 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.670865 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.670871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.671053 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.671225 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.671538 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:02:13.678267 2352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: Bootstrap starting.
I20260927 05:02:13.682492 2352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:13.683854 2352 log.cc:824] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:13.687150 2352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: No bootstrap required, opened a new log
I20260927 05:02:13.697980 2352 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:13.698382 2352 raft_consensus.cc:374] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:13.698529 2352 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb02cd22c6d426db761470825933ced, State: Initialized, Role: FOLLOWER
I20260927 05:02:13.699208 2352 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:13.700250 2528 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:13.700578 2528 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:13.701694 2352 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:02:13.711393 2539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:02:13.711602 2539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:02:13.710906 2285 master_runner.cc:186] Error getting master registration for 127.20.169.125:44923: OK, instance_id { permanent_uuid: "89fe910ccd35475db59a07bc90337052" instance_seqno: 1790485333011888 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:02:13.721984 2285 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:02:13.740790 2427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } attempt: 5
Sep 27 05:02:13 dist-test-slave-39f2 krb5kdc[2272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485332, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
W20260927 05:02:13.751960 2306 tablet.cc:2369] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:02:13.753357 2455 tablet.cc:2369] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:13.759341 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.741419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:38463 (local address 127.0.0.1:40834)
0927 05:02:13.741694 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:13.741714 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:13.741822 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:13.742220 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:13.742227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:13.742782 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:13.743021 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.743033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.744518 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.744524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.746018 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:13.746033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.748454 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.748461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:13.748682 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.749438 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:13.749463 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:13.752350 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:13.756492 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.756498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.756502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.757016 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.757429 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:13.757435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:13.757439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:13.757616 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:13.758257 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:13.758261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:13.758447 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 05:02:13.758645 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 05:02:13.758937 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 05:02:13.759452 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:13.741641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.20.169.124:38463)
0927 05:02:13.741838 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:13.741844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:13.741865 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:02:13.741924 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:13.741929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:13.742031 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:13.742157 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:13.743177 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.744366 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.747584 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:13.748281 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:13.748423 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:13.752559 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:13.752566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:13.752572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:13.752617 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:13.756347 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.757158 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.757164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.757168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.757310 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:13.757753 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:13.757758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:13.757762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:13.758132 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:13.758307 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:13.758793 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 05:02:13.759052 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:02:13.765913 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: Bootstrap starting.
I20260927 05:02:13.771440 2427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:13.772722 2427 log.cc:824] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:13.776038 2427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: No bootstrap required, opened a new log
I20260927 05:02:13.788286 2427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:13.788679 2427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:13.788851 2427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fe910ccd35475db59a07bc90337052, State: Initialized, Role: FOLLOWER
I20260927 05:02:13.789507 2427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:13.790483 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:13.790920 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:13.791735 2427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:13.811443 2360 master_runner.cc:263] No leader master found from master 89fe910ccd35475db59a07bc90337052
I20260927 05:02:13.811596 2360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 05:02:13.814026 2553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:02:13.814229 2553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 05:02:14.000779 2381 tablet.cc:2369] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:14.003459 2542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:14.003764 2542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:14.005623 2542 leader_election.cc:288] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deb02cd22c6d426db761470825933ced (127.20.169.126:37069), c454355fe74043c0aaca404263f2394c (127.20.169.124:38463)
I20260927 05:02:14.007217 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89fe910ccd35475db59a07bc90337052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb02cd22c6d426db761470825933ced" is_pre_election: true
I20260927 05:02:14.007824 2327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89fe910ccd35475db59a07bc90337052 in term 0.
I20260927 05:02:14.008679 2376 leader_election.cc:302] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89fe910ccd35475db59a07bc90337052, deb02cd22c6d426db761470825933ced; no voters:
I20260927 05:02:14.009212 2542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:02:14.009369 2542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:02:14.009488 2542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:14.013851 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89fe910ccd35475db59a07bc90337052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c454355fe74043c0aaca404263f2394c" is_pre_election: true
I20260927 05:02:14.014559 2476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89fe910ccd35475db59a07bc90337052 in term 0.
I20260927 05:02:14.019742 2542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:14.021081 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89fe910ccd35475db59a07bc90337052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb02cd22c6d426db761470825933ced"
I20260927 05:02:14.021313 2327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:14.025636 2327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89fe910ccd35475db59a07bc90337052 in term 1.
I20260927 05:02:14.027294 2376 leader_election.cc:302] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89fe910ccd35475db59a07bc90337052, deb02cd22c6d426db761470825933ced; no voters:
I20260927 05:02:14.030943 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89fe910ccd35475db59a07bc90337052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c454355fe74043c0aaca404263f2394c"
I20260927 05:02:14.031189 2476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:14.034830 2476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89fe910ccd35475db59a07bc90337052 in term 1.
I20260927 05:02:14.035630 2542 leader_election.cc:288] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [CANDIDATE]: Term 1 election: Requested vote from peers deb02cd22c6d426db761470825933ced (127.20.169.126:37069), c454355fe74043c0aaca404263f2394c (127.20.169.124:38463)
I20260927 05:02:14.036042 2542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:14.036497 2542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [term 1 LEADER]: Becoming Leader. State: Replica: 89fe910ccd35475db59a07bc90337052, State: Running, Role: LEADER
I20260927 05:02:14.037328 2542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } }
I20260927 05:02:14.057227 2555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fe910ccd35475db59a07bc90337052. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.058416 2555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:14.073422 2557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:14.075680 2557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:14.097249 2476 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [term 1 FOLLOWER]: Refusing update from remote peer 89fe910ccd35475db59a07bc90337052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:14.098425 2555 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:14.106096 2327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [term 1 FOLLOWER]: Refusing update from remote peer 89fe910ccd35475db59a07bc90337052: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:14.107071 2555 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:14.125243 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fe910ccd35475db59a07bc90337052. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.125564 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:14.127970 2558 mvcc.cc:205] Tried to move back new op lower bound from 7333827928452136960 to 7333827928228655104. Current Snapshot: MvccSnapshot[applied={T|T < 7333827928452136960}]
I20260927 05:02:14.134656 2528 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fe910ccd35475db59a07bc90337052. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.134979 2528 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:14.140357 2528 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.140615 2528 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:14.141278 2560 mvcc.cc:205] Tried to move back new op lower bound from 7333827928452136960 to 7333827928228655104. Current Snapshot: MvccSnapshot[applied={T|T < 7333827928452136960}]
I20260927 05:02:14.155260 2557 catalog_manager.cc:1347] Generated new cluster ID: 791222ab34f7496aa3637044cf2897f9
I20260927 05:02:14.155382 2557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:14.157302 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.157630 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:14.157826 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.158079 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:14.169224 2514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89fe910ccd35475db59a07bc90337052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deb02cd22c6d426db761470825933ced" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37069 } } peers { permanent_uuid: "89fe910ccd35475db59a07bc90337052" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 44923 } } peers { permanent_uuid: "c454355fe74043c0aaca404263f2394c" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 38463 } } }
I20260927 05:02:14.169543 2514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:02:14.195977 2557 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:02:14.197991 2557 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:14.210430 2557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89fe910ccd35475db59a07bc90337052: Generated new TSK 0
I20260927 05:02:14.211462 2557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:02:14.641914 2525 catalog_manager.cc:1259] Loaded cluster ID: 791222ab34f7496aa3637044cf2897f9
I20260927 05:02:14.642016 2525 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: loading cluster ID for follower catalog manager: success
I20260927 05:02:14.644515 2525 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: acquiring CA information for follower catalog manager: success
I20260927 05:02:14.646282 2525 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c454355fe74043c0aaca404263f2394c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:02:14.714071 2539 catalog_manager.cc:1259] Loaded cluster ID: 791222ab34f7496aa3637044cf2897f9
I20260927 05:02:14.714179 2539 catalog_manager.cc:1552] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: loading cluster ID for follower catalog manager: success
I20260927 05:02:14.716912 2539 catalog_manager.cc:1574] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: acquiring CA information for follower catalog manager: success
I20260927 05:02:14.718824 2539 catalog_manager.cc:1602] T 00000000000000000000000000000000 P deb02cd22c6d426db761470825933ced: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:02:14.793769 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2285
I20260927 05:02:14.795104 2347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:14.905817 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2285
I20260927 05:02:14.928233 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2360
I20260927 05:02:14.929636 2422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:15.037189 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2360
I20260927 05:02:15.060125 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2434
I20260927 05:02:15.061256 2496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:15.167685 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2434
2026-09-27T05:02:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4450 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:02:15 dist-test-slave-39f2 krb5kdc[2574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:02:15 dist-test-slave-39f2 krb5kdc[2574](info): set up 2 sockets
Sep 27 05:02:15 dist-test-slave-39f2 krb5kdc[2574](info): commencing operation
krb5kdc: starting...
W20260927 05:02:16.317144 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:02:16 dist-test-slave-39f2 krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:02:16.573660 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:36987
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:16.913379 2588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:16.913777 2588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:16.913962 2588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:16.921396 2588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:16.921510 2588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:16.921607 2588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:16.921729 2588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:16.921823 2588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:16.932214 2588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:36987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:16.933661 2588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:16.935061 2588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:16.947491 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:16.947561 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:16.947536 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:16.949036 2588 server_base.cc:1034] running on GCE node
I20260927 05:02:16.950029 2588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:16.951550 2588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:16.952878 2588 hybrid_clock.cc:648] HybridClock initialized: now 1790485336952768 us; error 73 us; skew 500 ppm
I20260927 05:02:16.953274 2588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:16 dist-test-slave-39f2 krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485336, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:16.956655 2588 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:16.958292 2588 webserver.cc:460] Webserver started at http://127.20.169.126:45881/ using document root <none> and password file <none>
I20260927 05:02:16.959012 2588 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:16.959158 2588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:16.959487 2588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:16.962620 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "9d156644a7a848ddb290de3d0d6efb4e"
format_stamp: "Formatted at 2026-09-27 05:02:16 on dist-test-slave-39f2"
server_key: "5a266c686e6cc8bda748929a33f4a9ea"
server_key_iv: "986ddb89b3049b347a56ec66b9293d5e"
server_key_version: "encryptionkey@0"
I20260927 05:02:16.963377 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "9d156644a7a848ddb290de3d0d6efb4e"
format_stamp: "Formatted at 2026-09-27 05:02:16 on dist-test-slave-39f2"
server_key: "5a266c686e6cc8bda748929a33f4a9ea"
server_key_iv: "986ddb89b3049b347a56ec66b9293d5e"
server_key_version: "encryptionkey@0"
I20260927 05:02:16.968922 2588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:02:16.972653 2603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:16.974195 2588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:16.974447 2588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "9d156644a7a848ddb290de3d0d6efb4e"
format_stamp: "Formatted at 2026-09-27 05:02:16 on dist-test-slave-39f2"
server_key: "5a266c686e6cc8bda748929a33f4a9ea"
server_key_iv: "986ddb89b3049b347a56ec66b9293d5e"
server_key_version: "encryptionkey@0"
I20260927 05:02:16.974665 2588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:17.004684 2588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:17.009070 2588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:17.009387 2588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:17.028581 2588 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:36987
I20260927 05:02:17.028666 2655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:36987 every 8 connection(s)
I20260927 05:02:17.030071 2588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:17.033906 2656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:17.039153 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2588
I20260927 05:02:17.039387 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:02:17.040149 21157 external_mini_cluster.cc:1428] Setting key 700c46424446e2978d62b8b019de83c0
I20260927 05:02:17.045725 2656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e: Bootstrap starting.
I20260927 05:02:17.049546 2656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:17.050644 2656 log.cc:824] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:17.053712 2656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e: No bootstrap required, opened a new log
Sep 27 05:02:17 dist-test-slave-39f2 krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:17.063658 2656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d156644a7a848ddb290de3d0d6efb4e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36987 } }
I20260927 05:02:17.064167 2656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:17.064286 2656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d156644a7a848ddb290de3d0d6efb4e, State: Initialized, Role: FOLLOWER
I20260927 05:02:17.065011 2656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d156644a7a848ddb290de3d0d6efb4e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36987 } }
I20260927 05:02:17.065357 2656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:17.065546 2656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:17.065727 2656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:17.066129 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:17.042289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.20.169.126:36987)
0927 05:02:17.043229 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:17.043244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:17.043288 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:02:17.045273 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:17.045299 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:17.045728 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:17.046135 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:17.048019 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:17.049559 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:17.051271 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:17.052046 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:17.052225 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:17.056846 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:17.056863 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:17.056884 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:17.056934 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:17.061133 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:17.061990 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:17.062000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:17.062011 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:17.062224 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:17.062659 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:17.062664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:17.062668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:17.063420 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:17.063701 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:17.064169 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 05:02:17.064639 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":431,"threads_started":1}
I20260927 05:02:17.068403 2656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d156644a7a848ddb290de3d0d6efb4e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36987 } }
I20260927 05:02:17.069168 2656 leader_election.cc:302] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d156644a7a848ddb290de3d0d6efb4e; no voters:
I20260927 05:02:17.069877 2656 leader_election.cc:288] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:02:17.070006 2661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:17.070613 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:02:17.072228 2656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:17.073129 2661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [term 1 LEADER]: Becoming Leader. State: Replica: 9d156644a7a848ddb290de3d0d6efb4e, State: Running, Role: LEADER
I20260927 05:02:17.073741 2661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d156644a7a848ddb290de3d0d6efb4e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36987 } }
I20260927 05:02:17.077121 2663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d156644a7a848ddb290de3d0d6efb4e. Latest consensus state: current_term: 1 leader_uuid: "9d156644a7a848ddb290de3d0d6efb4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d156644a7a848ddb290de3d0d6efb4e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36987 } } }
I20260927 05:02:17.077099 2662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d156644a7a848ddb290de3d0d6efb4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d156644a7a848ddb290de3d0d6efb4e" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36987 } } }
I20260927 05:02:17.077576 2663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:17.077576 2662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:17.079167 2673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:17.084079 2673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:17.092684 2673 catalog_manager.cc:1347] Generated new cluster ID: 0d69b3837130440cab574ec60ce81897
I20260927 05:02:17.092803 2673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:17.121544 2673 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:02:17.122946 2673 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:17.129552 2673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d156644a7a848ddb290de3d0d6efb4e: Generated new TSK 0
I20260927 05:02:17.130390 2673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:02:17.184602 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2588
I20260927 05:02:17.185866 2651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:17.286240 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2588
2026-09-27T05:02:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2110 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:02:17 dist-test-slave-39f2 krb5kdc[2684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:02:17 dist-test-slave-39f2 krb5kdc[2684](info): set up 2 sockets
Sep 27 05:02:17 dist-test-slave-39f2 krb5kdc[2684](info): commencing operation
krb5kdc: starting...
W20260927 05:02:18.396900 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:02:18 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:02:18.637913 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38855
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:18.965124 2698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:18.965413 2698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:18.965564 2698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:18.972926 2698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:18.973054 2698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:18.973136 2698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:18.973244 2698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:18.973313 2698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:18.983074 2698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:18.984395 2698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:18.985832 2698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:18.997843 2706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:18.997917 2704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:18.997892 2703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:18.998824 2698 server_base.cc:1034] running on GCE node
I20260927 05:02:18.999558 2698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:19.000963 2698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:19.002206 2698 hybrid_clock.cc:648] HybridClock initialized: now 1790485339002164 us; error 46 us; skew 500 ppm
I20260927 05:02:19.002591 2698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:19 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485339, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:19.009260 2698 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:19.011018 2698 webserver.cc:460] Webserver started at http://127.20.169.126:42555/ using document root <none> and password file <none>
I20260927 05:02:19.011749 2698 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:19.011909 2698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:19.012236 2698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:19.015412 2698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "791da222f14b4b6b93cb3eb7995e4d99"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "7059b3e51b12a722a87a1bda9309f699"
server_key_iv: "c82fcafe66ce3b569a6121d4b78a27fa"
server_key_version: "encryptionkey@0"
I20260927 05:02:19.016181 2698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "791da222f14b4b6b93cb3eb7995e4d99"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "7059b3e51b12a722a87a1bda9309f699"
server_key_iv: "c82fcafe66ce3b569a6121d4b78a27fa"
server_key_version: "encryptionkey@0"
I20260927 05:02:19.021503 2698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:02:19.025072 2713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:19.026496 2698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:02:19.026690 2698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "791da222f14b4b6b93cb3eb7995e4d99"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "7059b3e51b12a722a87a1bda9309f699"
server_key_iv: "c82fcafe66ce3b569a6121d4b78a27fa"
server_key_version: "encryptionkey@0"
I20260927 05:02:19.026898 2698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:19.047043 2698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:19.051549 2698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:19.051865 2698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:19.070797 2698 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:38855
I20260927 05:02:19.070839 2764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:38855 every 8 connection(s)
I20260927 05:02:19.072171 2698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:19.076020 2765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:19.080519 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2698
I20260927 05:02:19.080747 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:02:19.081604 21157 external_mini_cluster.cc:1428] Setting key 5a7399cf31388d08825031f0b923dcb3
I20260927 05:02:19.087886 2765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap starting.
I20260927 05:02:19.091022 2765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:19.092010 2765 log.cc:824] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:19.095559 2765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: No bootstrap required, opened a new log
Sep 27 05:02:19 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:19.104012 2765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:19.104391 2765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:19.104511 2765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791da222f14b4b6b93cb3eb7995e4d99, State: Initialized, Role: FOLLOWER
I20260927 05:02:19.105072 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:19.083780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.20.169.126:38855)
0927 05:02:19.084532 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:19.084549 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:19.084593 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:02:19.086603 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:19.086625 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:19.087002 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:19.087386 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:19.088959 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:19.090319 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:19.091716 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:19.092464 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:19.092681 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:19.097411 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:19.097418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:19.097433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:19.097475 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:19.100716 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:19.101471 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:19.101478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:19.101485 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:19.101657 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:19.102049 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:19.102056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:19.102059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:19.102646 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:19.102995 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:19.103404 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 05:02:19.103880 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":360,"threads_started":1}
I20260927 05:02:19.105195 2765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:19.105472 2765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:19.105643 2765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:19.105842 2765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:19.108266 2765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:19.108752 2765 leader_election.cc:302] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791da222f14b4b6b93cb3eb7995e4d99; no voters:
I20260927 05:02:19.109418 2765 leader_election.cc:288] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:02:19.109561 2770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:19.110298 2770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 LEADER]: Becoming Leader. State: Replica: 791da222f14b4b6b93cb3eb7995e4d99, State: Running, Role: LEADER
I20260927 05:02:19.110846 2770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:19.111506 2765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:19.113960 2772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791da222f14b4b6b93cb3eb7995e4d99. Latest consensus state: current_term: 1 leader_uuid: "791da222f14b4b6b93cb3eb7995e4d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } } }
I20260927 05:02:19.113950 2771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "791da222f14b4b6b93cb3eb7995e4d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } } }
I20260927 05:02:19.114382 2772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:19.114382 2771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:19.117957 2776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:19.123917 2776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:19.134593 2776 catalog_manager.cc:1347] Generated new cluster ID: 37d6ec98c74d46e28d642ec8ec6b5826
I20260927 05:02:19.134742 2776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:19.144456 2776 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:02:19.145835 2776 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:19.155505 2776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Generated new TSK 0
I20260927 05:02:19.156373 2776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:02:19.224287 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38855
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:19.594875 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:19.595130 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:19.595250 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:19.602532 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:19.602638 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:19.602802 2792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:02:19.612957 2792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:19.614259 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:19.615581 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:19.629453 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:19.629550 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:19.629522 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:19.630674 2792 server_base.cc:1034] running on GCE node
I20260927 05:02:19.631350 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:19.632332 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:19.633585 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790485339633527 us; error 47 us; skew 500 ppm
I20260927 05:02:19.633955 2792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:19 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485339, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:19.637518 2792 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:02:19.639050 2792 webserver.cc:460] Webserver started at http://127.20.169.65:38247/ using document root <none> and password file <none>
I20260927 05:02:19.639724 2792 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:19.639870 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:19.640190 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:19.643205 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "03cb9605b2a41b2d6a0ccd4d3c556e11"
server_key_iv: "f2aee4fdfb39e7c7f30c708d51564baa"
server_key_version: "encryptionkey@0"
I20260927 05:02:19.643999 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "03cb9605b2a41b2d6a0ccd4d3c556e11"
server_key_iv: "f2aee4fdfb39e7c7f30c708d51564baa"
server_key_version: "encryptionkey@0"
I20260927 05:02:19.649163 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:02:19.652616 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:19.654011 2792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:19.654208 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "03cb9605b2a41b2d6a0ccd4d3c556e11"
server_key_iv: "f2aee4fdfb39e7c7f30c708d51564baa"
server_key_version: "encryptionkey@0"
I20260927 05:02:19.654415 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:19.677178 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:19.681748 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:19.682070 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:19.683107 2792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:19.684528 2792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:19.684638 2792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:19.684799 2792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:19.684888 2792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 05:02:19.719681 2792 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:34393
I20260927 05:02:19.719733 2919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:34393 every 8 connection(s)
I20260927 05:02:19.721127 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:02:19.723109 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2792
I20260927 05:02:19.723376 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:02:19.724347 21157 external_mini_cluster.cc:1428] Setting key 29e1bc2f988e31074026e767167f443b
Sep 27 05:02:19 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485339, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:19.743039 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:19.724078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:38363 (local address 127.20.169.126:38855)
0927 05:02:19.724649 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:19.724654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:19.726000 (+ 1346us) server_negotiation.cc:407] Connection header received
0927 05:02:19.728203 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:19.728209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:19.728323 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:19.728451 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:19.730170 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:19.731010 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:19.732663 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:19.733146 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:19.733304 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:19.736938 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:19.736945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:19.736949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:19.736984 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:19.739669 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:19.740483 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:19.740494 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:19.740496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:19.740622 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:19.741170 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:19.741175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:19.741177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:19.741451 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:19.741621 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:19.742364 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 05:02:19.742597 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":295,"threads_started":1}
I20260927 05:02:19.744012 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:19.724851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:38363)
0927 05:02:19.725798 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:19.725850 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:19.727905 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:19.728695 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:19.728704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:19.729316 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:19.729976 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:19.730007 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:19.731162 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:19.731175 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:19.732499 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:19.732512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:19.733396 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:19.733406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:19.733656 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:19.734410 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:19.734435 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:19.736662 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:19.739828 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:19.739838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:19.739854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:19.740342 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:19.740739 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:19.740746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:19.740750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:19.741031 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:19.741600 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:19.741608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:19.741961 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 05:02:19.742238 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 05:02:19.742777 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":373,"threads_started":1}
I20260927 05:02:19.745805 2920 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38855
I20260927 05:02:19.746125 2920 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 05:02:19.746937 2920 heartbeater.cc:511] Master 127.20.169.126:38855 requested a full tablet report, sending...
I20260927 05:02:19.749184 2730 ts_manager.cc:194] Registered new tserver with Master: f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 05:02:19.751375 2730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:38363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:02:19.777619 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38855
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:20.103986 2927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:20.104265 2927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:20.104362 2927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:20.111761 2927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:20.111879 2927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:20.111994 2927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:02:20.121973 2927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:20.123190 2927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:20.124440 2927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:20.137892 2935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:20.138056 2932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:20.138003 2933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:20.139302 2927 server_base.cc:1034] running on GCE node
I20260927 05:02:20.139910 2927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:20.140873 2927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:20.142096 2927 hybrid_clock.cc:648] HybridClock initialized: now 1790485340142043 us; error 49 us; skew 500 ppm
I20260927 05:02:20.142417 2927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:20 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485340, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:20.145624 2927 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:02:20.146978 2927 webserver.cc:460] Webserver started at http://127.20.169.66:37133/ using document root <none> and password file <none>
I20260927 05:02:20.147584 2927 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:20.147686 2927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:20.147945 2927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:20.150808 2927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "55a0811ce05248d58240229f70f85888"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "fbc86294e5c4472dd918a58d21af1125"
server_key_iv: "1a4db13874555228e312bc6df144e17d"
server_key_version: "encryptionkey@0"
I20260927 05:02:20.151590 2927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "55a0811ce05248d58240229f70f85888"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "fbc86294e5c4472dd918a58d21af1125"
server_key_iv: "1a4db13874555228e312bc6df144e17d"
server_key_version: "encryptionkey@0"
I20260927 05:02:20.156895 2927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 05:02:20.160691 2942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:20.162143 2927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:20.162300 2927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "55a0811ce05248d58240229f70f85888"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "fbc86294e5c4472dd918a58d21af1125"
server_key_iv: "1a4db13874555228e312bc6df144e17d"
server_key_version: "encryptionkey@0"
I20260927 05:02:20.162501 2927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:20.187490 2927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:20.192132 2927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:20.192457 2927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:20.193512 2927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:20.194849 2927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:20.194921 2927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:20.195047 2927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:20.195096 2927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:20.231833 2927 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:40985
I20260927 05:02:20.231886 3054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:40985 every 8 connection(s)
I20260927 05:02:20.233220 2927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:02:20.241272 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 2927
I20260927 05:02:20.241472 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:02:20.241910 21157 external_mini_cluster.cc:1428] Setting key d1e248becfee6d07f3328fa70b853b0f
Sep 27 05:02:20 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485340, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:20.255124 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.235745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:53333 (local address 127.20.169.126:38855)
0927 05:02:20.235924 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.235928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.237110 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 05:02:20.238876 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.238880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.238953 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.239047 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:20.240569 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.241398 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.243356 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.243906 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.244053 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.247753 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:20.247763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:20.247772 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:20.247827 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:20.251702 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:20.252425 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:20.252433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:20.252440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:20.252620 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:20.253195 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:20.253199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:20.253201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:20.253480 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:20.253597 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.254601 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.254833 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:02:20.256539 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.236178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:53333)
0927 05:02:20.236955 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.237001 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.238593 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.239253 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.239260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.239827 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:20.240401 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.240419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.241572 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.241576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.243187 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.243202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.244139 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.244144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.244346 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.245053 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:20.245076 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:20.247510 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:20.251859 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:20.251866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:20.251878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:20.252298 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:20.252786 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:20.252793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:20.252799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:20.253077 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:20.253624 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:20.253633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:20.254131 (+ 498us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.254443 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.255056 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":261,"threads_started":1}
I20260927 05:02:20.258208 3055 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38855
I20260927 05:02:20.258536 3055 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:20.259383 3055 heartbeater.cc:511] Master 127.20.169.126:38855 requested a full tablet report, sending...
I20260927 05:02:20.260941 2730 ts_manager.cc:194] Registered new tserver with Master: 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985)
I20260927 05:02:20.262203 2730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:53333
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:02:20.299724 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38855
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:20.621544 3060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:20.621831 3060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:20.621948 3060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:20.629480 3060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:20.629594 3060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:20.629725 3060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:02:20.639745 3060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:20.641160 3060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:20.642505 3060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:20.656085 3065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:20.656129 3066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:20.656090 3068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:20.657126 3060 server_base.cc:1034] running on GCE node
I20260927 05:02:20.657894 3060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:20.658998 3060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:20.660238 3060 hybrid_clock.cc:648] HybridClock initialized: now 1790485340660157 us; error 90 us; skew 500 ppm
I20260927 05:02:20.660634 3060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:20 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485340, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:20.664121 3060 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:02:20.665833 3060 webserver.cc:460] Webserver started at http://127.20.169.67:37281/ using document root <none> and password file <none>
I20260927 05:02:20.666592 3060 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:20.666757 3060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:20.667079 3060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:20.670370 3060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "1a55edd7faf4ca4a288d157c46b17fa1"
server_key_iv: "afd216f71e64a35e03158110b3fce450"
server_key_version: "encryptionkey@0"
I20260927 05:02:20.671208 3060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "1a55edd7faf4ca4a288d157c46b17fa1"
server_key_iv: "afd216f71e64a35e03158110b3fce450"
server_key_version: "encryptionkey@0"
I20260927 05:02:20.676720 3060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:02:20.680418 3075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:20.681867 3060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 05:02:20.682046 3060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "1a55edd7faf4ca4a288d157c46b17fa1"
server_key_iv: "afd216f71e64a35e03158110b3fce450"
server_key_version: "encryptionkey@0"
I20260927 05:02:20.682251 3060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:20.699519 3060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:20.703898 3060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:20.704188 3060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:20.705202 3060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:20.706563 3060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:20.706667 3060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:20.706835 3060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:20.706908 3060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:20.742813 3060 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:43007
I20260927 05:02:20.742813 3188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:43007 every 8 connection(s)
I20260927 05:02:20.744199 3060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:02:20.752605 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3060
I20260927 05:02:20.752792 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:02:20.753311 21157 external_mini_cluster.cc:1428] Setting key 307fc7fdd0dee06002a73f566c9b558b
I20260927 05:02:20.754565 2920 heartbeater.cc:503] Master 127.20.169.126:38855 was elected leader, sending a full tablet report...
Sep 27 05:02:20 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485340, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:20.765506 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.746977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:59453 (local address 127.20.169.126:38855)
0927 05:02:20.747151 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.747155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.748286 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 05:02:20.750225 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.750231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.750348 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.750463 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:20.751947 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.752771 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.754566 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.755252 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.755388 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.759761 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:20.759766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:20.759770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:20.759805 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:20.762443 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:20.763210 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:20.763217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:20.763220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:20.763353 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:20.763783 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:20.763788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:20.763792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:20.764072 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:20.764228 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.764924 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.765148 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:02:20.766599 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.747375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:59453)
0927 05:02:20.748117 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.748158 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.749930 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.750700 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.750708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.751200 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:20.751776 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.751794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.752981 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.752985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.754414 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.754427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.755502 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.755510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.756379 (+ 869us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.757181 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:20.757204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:20.759538 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:20.762624 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:20.762633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:20.762651 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:20.763086 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:20.763464 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:20.763470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:20.763473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:20.763668 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:20.764201 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:20.764209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:20.764547 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.764849 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.765373 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":301,"threads_started":1}
I20260927 05:02:20.768093 3189 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38855
I20260927 05:02:20.768414 3189 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:20.769408 3189 heartbeater.cc:511] Master 127.20.169.126:38855 requested a full tablet report, sending...
I20260927 05:02:20.770852 2729 ts_manager.cc:194] Registered new tserver with Master: 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
I20260927 05:02:20.771760 2729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:59453
I20260927 05:02:20.781530 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:02:20.799211 2922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.786219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.20.169.126:38855)
0927 05:02:20.786446 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.786451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.786851 (+ 400us) server_negotiation.cc:407] Connection header received
0927 05:02:20.787001 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.787005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.787065 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.787145 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:20.788485 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.789257 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.790608 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.791055 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.791167 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.793797 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:20.793802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:20.793806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:20.793839 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:20.796747 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:20.797500 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:20.797504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:20.797507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:20.797657 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:20.798005 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:20.798008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:20.798010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:20.798336 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:20.798523 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.798792 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.798962 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:02:20.803752 2729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:02:20.810665 2729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:02:20.834015 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.823390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:40882)
0927 05:02:20.824455 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.824492 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.824687 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.825750 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.825760 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.825824 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:20.826501 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.826516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.828078 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.828086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.830764 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.830775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.832842 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.832855 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.833072 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.833123 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.833340 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.833545 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":34,"thread_start_us":686,"threads_started":1}
I20260927 05:02:20.834043 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.823586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.20.169.65:34393)
0927 05:02:20.824519 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.824526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.824596 (+ 70us) server_negotiation.cc:407] Connection header received
0927 05:02:20.824976 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.824987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.825257 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.825475 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:20.826729 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.827938 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.830979 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.832607 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.832773 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.833041 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.833432 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.833630 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":628,"threads_started":1}
I20260927 05:02:20.837741 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.826955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:43007 (local address 127.0.0.1:48252)
0927 05:02:20.828539 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.828556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.828655 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.830181 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.830191 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.830219 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:20.830680 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.830701 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.832354 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.832362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.835106 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.835119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.836787 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.836795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.837000 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.837046 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.837175 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.837307 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"mutex_wait_us":1,"thread_start_us":1287,"threads_started":1}
I20260927 05:02:20.837785 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.826418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.20.169.67:43007)
0927 05:02:20.829280 (+ 2862us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.829289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.829319 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:02:20.829411 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.829419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.829643 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.829897 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:20.830991 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.832232 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.835277 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.836567 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.836728 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.837014 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.837231 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.837400 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2674,"thread_start_us":2501,"threads_started":1}
I20260927 05:02:20.838621 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.824940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:40985 (local address 127.0.0.1:60946)
0927 05:02:20.825807 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.825825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.825980 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.830624 (+ 4644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.830629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.830668 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:20.831153 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.831168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.833265 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.833274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.835857 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.835873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.837918 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.837923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.838078 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.838107 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.838193 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.838309 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":583,"threads_started":1}
I20260927 05:02:20.838406 2855 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0896da84b8472dbc299989c0c73d68 (DEFAULT_TABLE table=test-table [id=170ad4ff35f6408098e653c29a8168eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:20.839351 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.825296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.20.169.66:40985)
0927 05:02:20.829749 (+ 4453us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.829776 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.829806 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:02:20.829907 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.829916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.830183 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.830402 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:20.831498 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.833078 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.836049 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.837474 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.837610 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.837788 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.838866 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.839027 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4300,"thread_start_us":4149,"threads_started":1}
I20260927 05:02:20.839999 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0896da84b8472dbc299989c0c73d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:20.841476 3124 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0896da84b8472dbc299989c0c73d68 (DEFAULT_TABLE table=test-table [id=170ad4ff35f6408098e653c29a8168eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:20.842860 3124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0896da84b8472dbc299989c0c73d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:20.844290 2990 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0896da84b8472dbc299989c0c73d68 (DEFAULT_TABLE table=test-table [id=170ad4ff35f6408098e653c29a8168eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:20.846266 2990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0896da84b8472dbc299989c0c73d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:20.851346 3207 tablet_bootstrap.cc:492] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Bootstrap starting.
I20260927 05:02:20.854761 3207 tablet_bootstrap.cc:654] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:20.855784 3207 log.cc:824] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:20.857708 3209 tablet_bootstrap.cc:492] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Bootstrap starting.
I20260927 05:02:20.858878 3207 tablet_bootstrap.cc:492] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: No bootstrap required, opened a new log
I20260927 05:02:20.859210 3207 ts_tablet_manager.cc:1397] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 05:02:20.861673 3209 tablet_bootstrap.cc:654] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:20.862929 3209 log.cc:824] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:20.864300 3210 tablet_bootstrap.cc:492] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Bootstrap starting.
I20260927 05:02:20.866129 3209 tablet_bootstrap.cc:492] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: No bootstrap required, opened a new log
I20260927 05:02:20.866416 3209 ts_tablet_manager.cc:1397] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:02:20.869732 3207 raft_consensus.cc:348] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.870179 3207 raft_consensus.cc:374] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:20.870321 3207 raft_consensus.cc:729] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c771a8f3fb41b990dc5c5ca2a96c46, State: Initialized, Role: FOLLOWER
I20260927 05:02:20.870931 3207 consensus_queue.cc:260] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.872587 3207 ts_tablet_manager.cc:1428] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 05:02:20.873157 3210 tablet_bootstrap.cc:654] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:20.874395 3210 log.cc:824] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:20.876008 3209 raft_consensus.cc:348] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.876367 3209 raft_consensus.cc:374] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:20.876477 3209 raft_consensus.cc:729] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5eba98ea7b437ea9b53fe8c36fa7ad, State: Initialized, Role: FOLLOWER
I20260927 05:02:20.877182 3209 consensus_queue.cc:260] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.877420 3210 tablet_bootstrap.cc:492] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: No bootstrap required, opened a new log
I20260927 05:02:20.877712 3210 ts_tablet_manager.cc:1397] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260927 05:02:20.878724 3189 heartbeater.cc:503] Master 127.20.169.126:38855 was elected leader, sending a full tablet report...
I20260927 05:02:20.878860 3209 ts_tablet_manager.cc:1428] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 05:02:20.885937 3210 raft_consensus.cc:348] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.886317 3210 raft_consensus.cc:374] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:20.886421 3210 raft_consensus.cc:729] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a0811ce05248d58240229f70f85888, State: Initialized, Role: FOLLOWER
I20260927 05:02:20.886991 3210 consensus_queue.cc:260] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.887948 3055 heartbeater.cc:503] Master 127.20.169.126:38855 was elected leader, sending a full tablet report...
I20260927 05:02:20.888422 3210 ts_tablet_manager.cc:1428] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 05:02:20.942685 3214 raft_consensus.cc:482] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:20.943022 3214 raft_consensus.cc:504] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.944849 3214 leader_election.cc:288] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393), 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985)
I20260927 05:02:20.953142 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.945900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:54161 (local address 127.20.169.66:40985)
0927 05:02:20.946093 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.946098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.947097 (+ 999us) server_negotiation.cc:407] Connection header received
0927 05:02:20.947381 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.947389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.947484 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.947592 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:20.948288 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.949041 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.950726 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.951981 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.952165 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.952259 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.952725 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.952889 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:02:20.953426 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.945666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:40985 (local address 127.20.169.67:54161)
0927 05:02:20.947000 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.947019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.947214 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.947629 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.947633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.947659 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:20.948131 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.948142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.949193 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.949197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.950575 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.950585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.952223 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.952231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.952484 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.952563 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.952699 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.952908 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":76,"thread_start_us":971,"threads_started":1}
I20260927 05:02:20.953701 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.945402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:55025 (local address 127.20.169.65:34393)
0927 05:02:20.945598 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.945606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.945625 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:02:20.945696 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.945703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.945781 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.945869 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:20.946539 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.947651 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.950789 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.952052 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.952222 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.952293 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.953020 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.953150 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:02:20.954136 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.945175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.20.169.67:55025)
0927 05:02:20.945383 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:20.945404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:20.945543 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:20.946172 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:20.946177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:20.946203 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:20.946404 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.946420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.947831 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.947843 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.950573 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:20.950594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.952384 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.952389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:20.952607 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.952655 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 05:02:20.952748 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:02:20.952890 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":112}
I20260927 05:02:20.954430 3010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0896da84b8472dbc299989c0c73d68" candidate_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a0811ce05248d58240229f70f85888" is_pre_election: true
I20260927 05:02:20.954985 3010 raft_consensus.cc:2454] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5eba98ea7b437ea9b53fe8c36fa7ad in term 0.
I20260927 05:02:20.955363 2875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0896da84b8472dbc299989c0c73d68" candidate_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" is_pre_election: true
I20260927 05:02:20.955638 3077 leader_election.cc:302] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad; no voters:
I20260927 05:02:20.955965 2875 raft_consensus.cc:2454] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b5eba98ea7b437ea9b53fe8c36fa7ad in term 0.
I20260927 05:02:20.956039 3214 raft_consensus.cc:2781] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:02:20.956185 3214 raft_consensus.cc:482] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:02:20.956298 3214 raft_consensus.cc:3037] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:20.958932 3214 raft_consensus.cc:504] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.959553 3214 leader_election.cc:288] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [CANDIDATE]: Term 1 election: Requested vote from peers f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393), 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985)
I20260927 05:02:20.960098 3010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0896da84b8472dbc299989c0c73d68" candidate_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a0811ce05248d58240229f70f85888"
I20260927 05:02:20.960139 2875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0896da84b8472dbc299989c0c73d68" candidate_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46"
I20260927 05:02:20.960309 3010 raft_consensus.cc:3037] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:20.960325 2875 raft_consensus.cc:3037] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:20.963603 3010 raft_consensus.cc:2454] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5eba98ea7b437ea9b53fe8c36fa7ad in term 1.
I20260927 05:02:20.963610 2875 raft_consensus.cc:2454] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b5eba98ea7b437ea9b53fe8c36fa7ad in term 1.
I20260927 05:02:20.963963 3077 leader_election.cc:302] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad; no voters:
I20260927 05:02:20.964253 3214 raft_consensus.cc:2781] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:20.964757 3214 raft_consensus.cc:686] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 LEADER]: Becoming Leader. State: Replica: 5b5eba98ea7b437ea9b53fe8c36fa7ad, State: Running, Role: LEADER
I20260927 05:02:20.965430 3214 consensus_queue.cc:237] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } }
I20260927 05:02:20.971698 2729 catalog_manager.cc:5495] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67). New cstate: current_term: 1 leader_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } health_report { overall_health: HEALTHY } } }
W20260927 05:02:20.973671 2921 tablet.cc:2369] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:02:20.985229 3056 tablet.cc:2369] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:20.989543 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:20.983587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.20.169.67:43007)
0927 05:02:20.983789 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:20.983795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:20.983991 (+ 196us) server_negotiation.cc:407] Connection header received
0927 05:02:20.984219 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:20.984223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:20.984296 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:20.984435 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:02:20.984834 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.985814 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.987351 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:20.987763 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:20.987905 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:20.988420 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:02:20.988973 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:02:20.989122 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:20.989190 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 05:02:20.989268 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 05:02:20.996296 3190 tablet.cc:2369] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:20.997494 3010 raft_consensus.cc:1261] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Refusing update from remote peer 5b5eba98ea7b437ea9b53fe8c36fa7ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:20.998019 2875 raft_consensus.cc:1261] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Refusing update from remote peer 5b5eba98ea7b437ea9b53fe8c36fa7ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:20.998457 3214 consensus_queue.cc:1035] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:20.998947 3217 consensus_queue.cc:1035] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:21.014446 3221 mvcc.cc:205] Tried to move back new op lower bound from 7333827956715065344 to 7333827956600098816. Current Snapshot: MvccSnapshot[applied={T|T < 7333827956715065344}]
I20260927 05:02:21.039611 2729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:21.039966 2729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:21.046892 2729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Sending DeleteTablet for 3 replicas of tablet 6c0896da84b8472dbc299989c0c73d68
I20260927 05:02:21.048281 2855 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0896da84b8472dbc299989c0c73d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:21 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:40882
I20260927 05:02:21.048269 2990 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0896da84b8472dbc299989c0c73d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:21 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:60946
I20260927 05:02:21.048952 2854 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0896da84b8472dbc299989c0c73d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:21 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:40882
I20260927 05:02:21.049273 3231 tablet_replica.cc:330] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: stopping tablet replica
I20260927 05:02:21.049702 3231 raft_consensus.cc:2229] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:21.049990 3124 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0896da84b8472dbc299989c0c73d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:21 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:48252
I20260927 05:02:21.050220 3231 raft_consensus.cc:2258] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:21.050900 3233 tablet_replica.cc:330] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: stopping tablet replica
I20260927 05:02:21.051271 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2792
I20260927 05:02:21.051648 3233 raft_consensus.cc:2229] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:02:21.052275 3233 raft_consensus.cc:2258] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:21.052878 3232 tablet_replica.cc:330] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: stopping tablet replica
I20260927 05:02:21.053352 3232 raft_consensus.cc:2229] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:21.053911 3232 raft_consensus.cc:2258] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:21.054646 3233 ts_tablet_manager.cc:1905] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:21.056087 3232 ts_tablet_manager.cc:1905] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:21.062785 3231 ts_tablet_manager.cc:1905] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:21.070282 3231 ts_tablet_manager.cc:1918] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:21.070277 3232 ts_tablet_manager.cc:1918] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:21.070452 3232 log.cc:1196] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/6c0896da84b8472dbc299989c0c73d68
I20260927 05:02:21.070559 3231 log.cc:1196] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/6c0896da84b8472dbc299989c0c73d68
I20260927 05:02:21.071676 3233 ts_tablet_manager.cc:1918] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:21.072376 3231 ts_tablet_manager.cc:1939] T 6c0896da84b8472dbc299989c0c73d68 P f2c771a8f3fb41b990dc5c5ca2a96c46: Deleting consensus metadata
I20260927 05:02:21.072862 3232 ts_tablet_manager.cc:1939] T 6c0896da84b8472dbc299989c0c73d68 P 55a0811ce05248d58240229f70f85888: Deleting consensus metadata
I20260927 05:02:21.074534 2715 catalog_manager.cc:4841] TS f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393): tablet 6c0896da84b8472dbc299989c0c73d68 (table test-table [id=170ad4ff35f6408098e653c29a8168eb]) successfully deleted
I20260927 05:02:21.074952 3233 log.cc:1196] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/6c0896da84b8472dbc299989c0c73d68
I20260927 05:02:21.075582 3233 ts_tablet_manager.cc:1939] T 6c0896da84b8472dbc299989c0c73d68 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Deleting consensus metadata
I20260927 05:02:21.076555 2715 catalog_manager.cc:4841] TS 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): tablet 6c0896da84b8472dbc299989c0c73d68 (table test-table [id=170ad4ff35f6408098e653c29a8168eb]) successfully deleted
I20260927 05:02:21.077297 2715 catalog_manager.cc:4841] TS 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): tablet 6c0896da84b8472dbc299989c0c73d68 (table test-table [id=170ad4ff35f6408098e653c29a8168eb]) successfully deleted
W20260927 05:02:21.077929 2715 catalog_manager.cc:4808] TS f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393): delete failed for tablet 6c0896da84b8472dbc299989c0c73d68 because the tablet was not found. No further retry: Not found: Tablet not found: 6c0896da84b8472dbc299989c0c73d68
Sep 27 05:02:21 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:02:21.090117 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:21.052527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.20.169.65:34393)
0927 05:02:21.052731 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:21.052740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:21.058824 (+ 6084us) server_negotiation.cc:407] Connection header received
0927 05:02:21.059657 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:21.059663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:21.059745 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:21.059824 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:21.065804 (+ 5980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:21.066995 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:21.070649 (+ 3654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:21.071272 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:21.071407 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:21.082041 (+ 10634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:21.082058 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:21.082075 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:21.082124 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:21.086342 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:21.087305 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:21.087315 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:21.087319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:21.087491 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:21.087965 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:21.087970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:21.087973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:21.088431 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:21.088738 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:21.089417 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 05:02:21.089672 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:02:21.091917 2915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:21.202028 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2792
I20260927 05:02:21.222594 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2927
Sep 27 05:02:21 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:02:21.240145 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:21.223506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.20.169.66:40985)
0927 05:02:21.223688 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:21.223693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:21.223714 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:02:21.223839 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:21.223845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:21.223933 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:21.224055 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:21.225444 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:21.226173 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:21.227517 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:21.227949 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:21.228088 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:21.232976 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:21.232992 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:21.233013 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:21.233068 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:21.237062 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:21.237859 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:21.237868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:21.237872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:21.238013 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:21.238406 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:21.238412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:21.238415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:21.238886 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:21.239151 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:21.239504 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 05:02:21.239736 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:02:21.240860 3050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:21.329946 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2927
I20260927 05:02:21.349099 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 3060
Sep 27 05:02:21 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:02:21.368251 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:21.350053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.20.169.67:43007)
0927 05:02:21.350279 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:21.350285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:21.350308 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:02:21.350459 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:21.350465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:21.350577 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:21.350708 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:21.351998 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:21.353219 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:21.354658 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:21.355363 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:21.355503 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:21.360309 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:21.360324 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:21.360345 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:21.360405 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:21.364615 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:21.365577 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:21.365588 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:21.365592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:21.365805 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:21.366307 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:21.366312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:21.366316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:21.366755 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:21.367084 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:21.367532 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 05:02:21.367786 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:02:21.369489 3184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:21.503520 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 3060
I20260927 05:02:21.524430 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 2698
I20260927 05:02:21.525764 2760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:21.637621 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 2698
I20260927 05:02:21.660002 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38855
--webserver_interface=127.20.169.126
--webserver_port=42555
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:21.994028 3239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:21.994370 3239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:21.994526 3239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:22.001745 3239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:22.001884 3239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:22.001993 3239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:22.002094 3239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:22.002180 3239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:22.011986 3239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:22.013432 3239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:22.014791 3239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:22.039635 3245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:22.039615 3248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:22.039667 3246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:22.040601 3239 server_base.cc:1034] running on GCE node
I20260927 05:02:22.041513 3239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:22.043110 3239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:22.044400 3239 hybrid_clock.cc:648] HybridClock initialized: now 1790485342044303 us; error 38 us; skew 500 ppm
I20260927 05:02:22.044864 3239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:22 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:22.048439 3239 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:22.050091 3239 webserver.cc:460] Webserver started at http://127.20.169.126:42555/ using document root <none> and password file <none>
I20260927 05:02:22.050800 3239 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:22.050921 3239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:22.057118 3239 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:02:22.060328 3255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:22.061908 3239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:22.062095 3239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "791da222f14b4b6b93cb3eb7995e4d99"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "7059b3e51b12a722a87a1bda9309f699"
server_key_iv: "c82fcafe66ce3b569a6121d4b78a27fa"
server_key_version: "encryptionkey@0"
I20260927 05:02:22.063191 3239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:22.090869 3239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:22.095620 3239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:22.095937 3239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:22.115568 3239 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:38855
I20260927 05:02:22.115592 3306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:38855 every 8 connection(s)
I20260927 05:02:22.117175 3239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:22.124218 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3239
I20260927 05:02:22.125308 3307 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:02:22.125347 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:34393
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:38855
--webserver_port=38247
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:22.130810 3307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap starting.
I20260927 05:02:22.152206 3307 log.cc:824] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:22.164902 3307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:02:22.165441 3307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap complete.
I20260927 05:02:22.175017 3307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:22.175858 3307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791da222f14b4b6b93cb3eb7995e4d99, State: Initialized, Role: FOLLOWER
I20260927 05:02:22.176496 3307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:22.176719 3307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:22.176875 3307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:22.177059 3307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:02:22.181094 3307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:22.181615 3307 leader_election.cc:302] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791da222f14b4b6b93cb3eb7995e4d99; no voters:
I20260927 05:02:22.182258 3307 leader_election.cc:288] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:02:22.182408 3311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:02:22.183542 3311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 LEADER]: Becoming Leader. State: Replica: 791da222f14b4b6b93cb3eb7995e4d99, State: Running, Role: LEADER
I20260927 05:02:22.184159 3311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:22.184681 3307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:22.187124 3312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "791da222f14b4b6b93cb3eb7995e4d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } } }
I20260927 05:02:22.187484 3312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:22.187180 3313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791da222f14b4b6b93cb3eb7995e4d99. Latest consensus state: current_term: 2 leader_uuid: "791da222f14b4b6b93cb3eb7995e4d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } } }
I20260927 05:02:22.187597 3313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:22.188707 3318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:22.216250 3318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:22.218490 3318 catalog_manager.cc:1259] Loaded cluster ID: 37d6ec98c74d46e28d642ec8ec6b5826
I20260927 05:02:22.218935 3318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:22.223328 3318 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 05:02:22.250144 3318 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 05:02:22.265429 3318 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:22.267241 3318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Loaded TSK: 0
I20260927 05:02:22.268793 3318 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:02:22.488199 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:22.488480 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:22.488622 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:22.496145 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:22.496248 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:22.496423 3309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:02:22.506757 3309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:34393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=38247
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:22.508066 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:22.509505 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:22.524633 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:22.524648 3334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:22.526002 3309 server_base.cc:1034] running on GCE node
W20260927 05:02:22.526122 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:22.526930 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:22.527992 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:22.529232 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790485342529162 us; error 54 us; skew 500 ppm
I20260927 05:02:22.529579 3309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:22 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:22.533080 3309 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:02:22.534742 3309 webserver.cc:460] Webserver started at http://127.20.169.65:38247/ using document root <none> and password file <none>
I20260927 05:02:22.535771 3309 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:22.535965 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:22.542002 3309 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:02:22.544988 3344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:22.546516 3309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:02:22.546708 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "03cb9605b2a41b2d6a0ccd4d3c556e11"
server_key_iv: "f2aee4fdfb39e7c7f30c708d51564baa"
server_key_version: "encryptionkey@0"
I20260927 05:02:22.547703 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:22.569165 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:22.574028 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:22.574337 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:22.575515 3309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:22.577029 3309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:22.577157 3309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:22.577329 3309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:22.577425 3309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:22.613857 3309 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:34393
I20260927 05:02:22.613915 3458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:34393 every 8 connection(s)
I20260927 05:02:22.615358 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:02:22.622555 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3309
I20260927 05:02:22.623322 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:40985
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:38855
--webserver_port=37133
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:02:22 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:22.642503 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:22.618550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:59525)
0927 05:02:22.619406 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:22.619458 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:22.621309 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:22.622750 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:22.622761 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:22.623537 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:22.624345 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:22.624374 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:22.626105 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:22.626113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:22.628306 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:22.628325 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:22.629462 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:22.629471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:22.629795 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:22.630581 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:22.630621 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:22.633486 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:22.638199 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:22.638205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:22.638216 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:22.638662 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:22.639152 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:22.639155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:22.639157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:22.639372 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:22.640375 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:22.640386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:22.640696 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 05:02:22.641032 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 05:02:22.641438 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":358,"threads_started":1}
I20260927 05:02:22.643065 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:22.618274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:59525 (local address 127.20.169.126:38855)
0927 05:02:22.618855 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:22.618868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:22.619588 (+ 720us) server_negotiation.cc:407] Connection header received
0927 05:02:22.621830 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:22.621843 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:22.622202 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:22.622508 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:22.624604 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:22.625935 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:22.628476 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:22.629218 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:22.629371 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:22.633766 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:22.633778 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:22.633800 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:22.633850 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:22.638051 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:22.638803 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:22.638814 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:22.638825 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:22.639034 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:22.639484 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:22.639491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:22.639497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:22.640146 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:22.640356 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:22.641268 (+ 912us) server_negotiation.cc:299] Negotiation successful
0927 05:02:22.641754 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":234,"threads_started":1}
I20260927 05:02:22.646050 3459 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38855
I20260927 05:02:22.646404 3459 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:22.647354 3459 heartbeater.cc:511] Master 127.20.169.126:38855 requested a full tablet report, sending...
I20260927 05:02:22.649772 3272 ts_manager.cc:194] Registered new tserver with Master: f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393)
I20260927 05:02:22.651367 3272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:59525
W20260927 05:02:22.951804 3463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:22.952126 3463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:22.952222 3463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:22.959328 3463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:22.959473 3463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:22.959659 3463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:02:22.969421 3463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:40985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=37133
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:22.970690 3463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:22.972016 3463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:22.985534 3468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:22.985534 3469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:22.985569 3471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:22.986428 3463 server_base.cc:1034] running on GCE node
I20260927 05:02:22.987036 3463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:22.987936 3463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:22.989126 3463 hybrid_clock.cc:648] HybridClock initialized: now 1790485342989074 us; error 44 us; skew 500 ppm
I20260927 05:02:22.989426 3463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:22 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:22.992507 3463 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:02:22.993945 3463 webserver.cc:460] Webserver started at http://127.20.169.66:37133/ using document root <none> and password file <none>
I20260927 05:02:22.994553 3463 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:22.994663 3463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:22.999837 3463 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 05:02:23.002513 3478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:23.003818 3463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:02:23.003988 3463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "55a0811ce05248d58240229f70f85888"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "fbc86294e5c4472dd918a58d21af1125"
server_key_iv: "1a4db13874555228e312bc6df144e17d"
server_key_version: "encryptionkey@0"
I20260927 05:02:23.004988 3463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:23.022152 3463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:23.026584 3463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:23.026885 3463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:23.027915 3463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:23.029332 3463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:23.029451 3463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:23.029585 3463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:23.029680 3463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:23.064432 3463 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:40985
I20260927 05:02:23.064486 3591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:40985 every 8 connection(s)
I20260927 05:02:23.065918 3463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:02:23.071698 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3463
I20260927 05:02:23.072492 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:43007
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:38855
--webserver_port=37281
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:02:23 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485342, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:23.089452 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.068618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40337 (local address 127.20.169.126:38855)
0927 05:02:23.068788 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.068792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.070092 (+ 1300us) server_negotiation.cc:407] Connection header received
0927 05:02:23.072282 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.072287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.072371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.072472 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:23.074693 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.075414 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.078028 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.078454 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.078553 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.083169 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:23.083176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:23.083182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:23.083232 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:23.086218 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.087136 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.087140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.087142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.087264 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.087757 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.087760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.087762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.088022 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:23.088141 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.089030 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.089204 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:02:23.091058 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.069242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:40337)
0927 05:02:23.069922 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.069961 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.071962 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.072784 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.072798 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.073630 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:23.074498 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.074527 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.075580 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.075589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.077869 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.077887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.078685 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.078694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.079023 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.079753 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:23.079787 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:23.082898 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:23.086347 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:23.086358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:23.086379 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:23.087009 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:23.087384 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:23.087392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:23.087398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:23.087645 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:23.088157 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:23.088167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:23.088629 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.088935 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.089540 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":266,"threads_started":1}
I20260927 05:02:23.092736 3592 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38855
I20260927 05:02:23.093103 3592 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:23.093865 3592 heartbeater.cc:511] Master 127.20.169.126:38855 requested a full tablet report, sending...
I20260927 05:02:23.095384 3272 ts_manager.cc:194] Registered new tserver with Master: 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985)
I20260927 05:02:23.096742 3272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:40337
W20260927 05:02:23.410102 3595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:23.410375 3595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:23.410468 3595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:23.417485 3595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:23.417588 3595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:23.417696 3595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:02:23.427394 3595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:43007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=37281
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:23.428606 3595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:23.429930 3595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:23.442950 3600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:23.442912 3601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:23.442898 3603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:23.444020 3595 server_base.cc:1034] running on GCE node
I20260927 05:02:23.444629 3595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:23.445614 3595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:23.446837 3595 hybrid_clock.cc:648] HybridClock initialized: now 1790485343446795 us; error 35 us; skew 500 ppm
I20260927 05:02:23.447170 3595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:23 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485343, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:23.450529 3595 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:02:23.452097 3595 webserver.cc:460] Webserver started at http://127.20.169.67:37281/ using document root <none> and password file <none>
I20260927 05:02:23.452867 3595 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:23.453043 3595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:23.458485 3595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:02:23.461404 3610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:23.462899 3595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:02:23.463192 3595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "1a55edd7faf4ca4a288d157c46b17fa1"
server_key_iv: "afd216f71e64a35e03158110b3fce450"
server_key_version: "encryptionkey@0"
I20260927 05:02:23.464267 3595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:23.494518 3595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:23.498934 3595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:23.499257 3595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:23.500252 3595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:23.501619 3595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:23.501739 3595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:23.501873 3595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:23.501960 3595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:23.536068 3595 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:43007
I20260927 05:02:23.536166 3723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:43007 every 8 connection(s)
I20260927 05:02:23.537602 3595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:02:23.540755 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3595
Sep 27 05:02:23 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485343, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:23.557395 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.541866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.20.169.126:38855)
0927 05:02:23.542582 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.542590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.542950 (+ 360us) server_negotiation.cc:407] Connection header received
0927 05:02:23.543158 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.543168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.543231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.543353 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:23.545112 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.545891 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.547330 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.547739 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.547879 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.551005 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:23.551009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:23.551013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:23.551042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:23.554416 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.555260 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.555264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.555266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.555407 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.555814 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.555816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.555818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.556204 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:23.556479 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.556727 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.557007 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":493,"threads_started":1}
I20260927 05:02:23.559530 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.540210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:42551 (local address 127.20.169.126:38855)
0927 05:02:23.540426 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.540430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.542052 (+ 1622us) server_negotiation.cc:407] Connection header received
0927 05:02:23.544144 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.544151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.544225 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.544346 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:23.545819 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.546601 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.548099 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.548726 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.548910 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.552611 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:23.552616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:23.552619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:23.552684 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:23.555833 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.556770 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.556775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.556778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.556996 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.557526 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.557534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.557539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.557864 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:23.557996 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.558885 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.559118 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 05:02:23.560565 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.540914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:42551)
0927 05:02:23.541858 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.541913 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.543891 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.544535 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.544542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.545073 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:23.545645 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.545662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.546764 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.546769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.547956 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.547967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.548970 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.548979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.549239 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.549869 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:23.549890 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:23.552336 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:23.555976 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:23.555986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:23.556003 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:23.556640 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:23.557122 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:23.557127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:23.557131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:23.557372 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:23.558051 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:23.558062 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:23.558421 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.558668 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.559168 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":319,"threads_started":1}
I20260927 05:02:23.562041 3724 heartbeater.cc:347] Connected to a master server at 127.20.169.126:38855
I20260927 05:02:23.562362 3724 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:23.563050 3724 heartbeater.cc:511] Master 127.20.169.126:38855 requested a full tablet report, sending...
I20260927 05:02:23.564429 3272 ts_manager.cc:194] Registered new tserver with Master: 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
I20260927 05:02:23.565301 3272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:42551
I20260927 05:02:23.572659 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:02:23.579205 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.573575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.20.169.126:38855)
0927 05:02:23.573737 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.573741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.574202 (+ 461us) server_negotiation.cc:407] Connection header received
0927 05:02:23.574391 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.574394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.574523 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.574640 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:02:23.574997 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.575689 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.577238 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.577709 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.577824 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.578226 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:02:23.578743 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:02:23.578859 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.578945 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.579006 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:02:23.580041 3272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:02:23.588249 3272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:02:23.620299 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.601449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:43007 (local address 127.0.0.1:54132)
0927 05:02:23.607526 (+ 6077us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.607551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.607706 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.609047 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.609059 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.609103 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:23.609625 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.609642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.612373 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.612382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.615639 (+ 3257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.615664 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.618897 (+ 3233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.618906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.619230 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.619307 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.619454 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.619756 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5913,"thread_start_us":5739,"threads_started":1}
I20260927 05:02:23.621698 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.602024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.20.169.67:43007)
0927 05:02:23.602982 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.602996 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.607866 (+ 4870us) server_negotiation.cc:407] Connection header received
0927 05:02:23.608019 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.608031 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.608366 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.608594 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:23.609912 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.612151 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.615917 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.618229 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.618402 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.618722 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.620955 (+ 2233us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.621225 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":482,"threads_started":1}
I20260927 05:02:23.628407 3659 tablet_service.cc:1467] Processing CreateTablet for tablet 82ec7a3a7bc84a8fb19773a4b141dd66 (DEFAULT_TABLE table=test-table [id=32a7021903114e109df2b49ca780253e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:23.630076 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ec7a3a7bc84a8fb19773a4b141dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:23.630352 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.608103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.20.169.65:34393)
0927 05:02:23.621930 (+ 13827us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.621940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.621972 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:02:23.622064 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.622073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.622371 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.622589 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:23.623089 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.625499 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.627802 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.629251 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.629401 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.629640 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.629841 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.630010 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13644,"thread_start_us":14913,"threads_started":1}
I20260927 05:02:23.630890 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.607528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52550)
0927 05:02:23.620571 (+ 13043us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.620589 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.620704 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.622597 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.622605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.622643 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:23.622871 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.622885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.625700 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.625707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.627628 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.627644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.629449 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.629455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.629638 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.629673 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.629769 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.629933 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12950,"thread_start_us":16481,"threads_started":1}
I20260927 05:02:23.633018 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.600320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.20.169.66:40985)
0927 05:02:23.625923 (+ 25603us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.625935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.625967 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:02:23.626067 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.626088 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.626335 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.626544 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:23.627170 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.628324 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.630528 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.631943 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.632103 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.632291 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.632538 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.632693 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25433,"thread_start_us":25301,"threads_started":1}
I20260927 05:02:23.633019 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.599935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:40985 (local address 127.0.0.1:43878)
0927 05:02:23.601199 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.601274 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.601473 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.626558 (+ 25085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.626563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.626591 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:23.627000 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.627012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.628466 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.628474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.630322 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.630338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.632159 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.632165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.632336 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.632374 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.632486 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.632617 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":1,"thread_start_us":737,"threads_started":1}
I20260927 05:02:23.635114 3392 tablet_service.cc:1467] Processing CreateTablet for tablet 82ec7a3a7bc84a8fb19773a4b141dd66 (DEFAULT_TABLE table=test-table [id=32a7021903114e109df2b49ca780253e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:23.636607 3392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ec7a3a7bc84a8fb19773a4b141dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:23.636446 3526 tablet_service.cc:1467] Processing CreateTablet for tablet 82ec7a3a7bc84a8fb19773a4b141dd66 (DEFAULT_TABLE table=test-table [id=32a7021903114e109df2b49ca780253e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:23.637713 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ec7a3a7bc84a8fb19773a4b141dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:23.645915 3740 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Bootstrap starting.
I20260927 05:02:23.649808 3741 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Bootstrap starting.
I20260927 05:02:23.650571 3740 tablet_bootstrap.cc:654] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:23.651706 3742 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Bootstrap starting.
I20260927 05:02:23.652200 3740 log.cc:824] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:23.653685 3741 tablet_bootstrap.cc:654] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:23.654665 3741 log.cc:824] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:23.654839 3459 heartbeater.cc:503] Master 127.20.169.126:38855 was elected leader, sending a full tablet report...
I20260927 05:02:23.655254 3742 tablet_bootstrap.cc:654] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:23.655440 3740 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: No bootstrap required, opened a new log
I20260927 05:02:23.655716 3740 ts_tablet_manager.cc:1397] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 05:02:23.656562 3742 log.cc:824] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:23.662668 3742 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: No bootstrap required, opened a new log
I20260927 05:02:23.662916 3741 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: No bootstrap required, opened a new log
I20260927 05:02:23.663090 3742 ts_tablet_manager.cc:1397] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 05:02:23.663159 3741 ts_tablet_manager.cc:1397] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260927 05:02:23.667361 3740 raft_consensus.cc:348] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.667856 3740 raft_consensus.cc:374] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:23.667959 3740 raft_consensus.cc:729] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5eba98ea7b437ea9b53fe8c36fa7ad, State: Initialized, Role: FOLLOWER
I20260927 05:02:23.668625 3740 consensus_queue.cc:260] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.669896 3724 heartbeater.cc:503] Master 127.20.169.126:38855 was elected leader, sending a full tablet report...
I20260927 05:02:23.670499 3740 ts_tablet_manager.cc:1428] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260927 05:02:23.671937 3742 raft_consensus.cc:348] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.672287 3742 raft_consensus.cc:374] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:23.672385 3742 raft_consensus.cc:729] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c771a8f3fb41b990dc5c5ca2a96c46, State: Initialized, Role: FOLLOWER
I20260927 05:02:23.673007 3742 consensus_queue.cc:260] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.673170 3741 raft_consensus.cc:348] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.673581 3741 raft_consensus.cc:374] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:23.673695 3741 raft_consensus.cc:729] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a0811ce05248d58240229f70f85888, State: Initialized, Role: FOLLOWER
I20260927 05:02:23.674345 3741 consensus_queue.cc:260] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.674892 3742 ts_tablet_manager.cc:1428] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 05:02:23.676927 3592 heartbeater.cc:503] Master 127.20.169.126:38855 was elected leader, sending a full tablet report...
I20260927 05:02:23.677590 3741 ts_tablet_manager.cc:1428] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260927 05:02:23.745859 3747 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:23.746206 3747 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.747943 3747 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
I20260927 05:02:23.756501 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.748556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:45469 (local address 127.20.169.66:40985)
0927 05:02:23.748752 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.748756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.749546 (+ 790us) server_negotiation.cc:407] Connection header received
0927 05:02:23.749864 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.749869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.749951 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.750070 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:23.750784 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.751609 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.754343 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.755574 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.755738 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.755832 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.756094 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.756236 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:02:23.756671 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.748509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:40985 (local address 127.20.169.65:45469)
0927 05:02:23.749399 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.749426 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.749589 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.750239 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.750243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.750312 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:23.750631 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.750640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.751730 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.751738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.754119 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.754134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.755803 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.755808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.755925 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.755950 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.756036 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.756158 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":46,"thread_start_us":519,"threads_started":1}
I20260927 05:02:23.757181 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.749613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:43007 (local address 127.20.169.65:46697)
0927 05:02:23.750317 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:23.750331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:23.750505 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:23.750890 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:23.750895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:23.750919 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:23.751331 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.751342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.752583 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.752588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.754587 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:23.754597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.756274 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.756280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:23.756458 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.756487 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:02:23.756691 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 05:02:23.756934 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":4,"thread_start_us":377,"threads_started":1}
I20260927 05:02:23.757503 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.749896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:46697 (local address 127.20.169.67:43007)
0927 05:02:23.750124 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.750129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.750434 (+ 305us) server_negotiation.cc:407] Connection header received
0927 05:02:23.750669 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.750675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.750781 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.750876 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:23.751504 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.752432 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.754754 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.756045 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.756219 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.756294 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.757074 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.757225 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:02:23.757895 3547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ec7a3a7bc84a8fb19773a4b141dd66" candidate_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a0811ce05248d58240229f70f85888" is_pre_election: true
I20260927 05:02:23.758447 3547 raft_consensus.cc:2454] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c771a8f3fb41b990dc5c5ca2a96c46 in term 0.
I20260927 05:02:23.758533 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ec7a3a7bc84a8fb19773a4b141dd66" candidate_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" is_pre_election: true
I20260927 05:02:23.759073 3679 raft_consensus.cc:2454] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c771a8f3fb41b990dc5c5ca2a96c46 in term 0.
I20260927 05:02:23.759181 3346 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a0811ce05248d58240229f70f85888, f2c771a8f3fb41b990dc5c5ca2a96c46; no voters:
I20260927 05:02:23.759608 3747 raft_consensus.cc:2781] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:02:23.759821 3747 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:02:23.759940 3747 raft_consensus.cc:3037] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:23.762171 3747 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.762749 3747 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 1 election: Requested vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
I20260927 05:02:23.763327 3547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ec7a3a7bc84a8fb19773a4b141dd66" candidate_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a0811ce05248d58240229f70f85888"
I20260927 05:02:23.763412 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ec7a3a7bc84a8fb19773a4b141dd66" candidate_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad"
I20260927 05:02:23.763541 3547 raft_consensus.cc:3037] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:23.763661 3679 raft_consensus.cc:3037] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:23.765904 3547 raft_consensus.cc:2454] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c771a8f3fb41b990dc5c5ca2a96c46 in term 1.
I20260927 05:02:23.765969 3679 raft_consensus.cc:2454] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c771a8f3fb41b990dc5c5ca2a96c46 in term 1.
I20260927 05:02:23.766307 3346 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a0811ce05248d58240229f70f85888, f2c771a8f3fb41b990dc5c5ca2a96c46; no voters:
I20260927 05:02:23.766571 3747 raft_consensus.cc:2781] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:23.767001 3747 raft_consensus.cc:686] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 LEADER]: Becoming Leader. State: Replica: f2c771a8f3fb41b990dc5c5ca2a96c46, State: Running, Role: LEADER
I20260927 05:02:23.767520 3747 consensus_queue.cc:237] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:23.772519 3272 catalog_manager.cc:5495] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65). New cstate: current_term: 1 leader_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } health_report { overall_health: HEALTHY } } }
W20260927 05:02:23.789243 3725 tablet.cc:2369] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:23.808219 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.802352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.20.169.65:34393)
0927 05:02:23.802541 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.802546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.802563 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:02:23.802738 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.802742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.802841 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.802970 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:02:23.803389 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.804355 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.806063 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.806516 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.806621 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.807110 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:02:23.807659 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:02:23.807833 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.807887 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.807957 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:02:23.817030 3547 raft_consensus.cc:1261] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Refusing update from remote peer f2c771a8f3fb41b990dc5c5ca2a96c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:23.817122 3679 raft_consensus.cc:1261] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Refusing update from remote peer f2c771a8f3fb41b990dc5c5ca2a96c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 05:02:23.817874 3593 tablet.cc:2369] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:23.818102 3751 consensus_queue.cc:1035] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:23.818567 3747 consensus_queue.cc:1035] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:23.843834 3272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:23.844115 3272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:23.847432 3272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Sending DeleteTablet for 3 replicas of tablet 82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:23.848588 3526 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ec7a3a7bc84a8fb19773a4b141dd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:23 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:43878
I20260927 05:02:23.848791 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 3309
I20260927 05:02:23.848744 3659 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ec7a3a7bc84a8fb19773a4b141dd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:23 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:54132
I20260927 05:02:23.849346 3392 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ec7a3a7bc84a8fb19773a4b141dd66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:23 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:52550
I20260927 05:02:23.882563 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:23.850358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.20.169.65:34393)
0927 05:02:23.850575 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:23.850582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:23.850602 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:02:23.850663 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:23.850671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:23.850761 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:23.850883 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:23.853311 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.854760 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.857089 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:23.858039 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:23.858177 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:23.862440 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:23.862458 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:23.862482 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:23.862544 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:23.869710 (+ 7166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.872608 (+ 2898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.872623 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.872628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.872899 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:23.873399 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:23.873409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:23.873413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:23.874025 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:23.874309 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:23.875279 (+ 970us) server_negotiation.cc:299] Negotiation successful
0927 05:02:23.875714 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 05:02:23.883435 3766 tablet_replica.cc:330] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: stopping tablet replica
I20260927 05:02:23.883527 3454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:23.883873 3765 tablet_replica.cc:330] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: stopping tablet replica
I20260927 05:02:23.884409 3765 raft_consensus.cc:2229] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:23.884642 3764 tablet_replica.cc:330] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: stopping tablet replica
I20260927 05:02:23.885134 3765 raft_consensus.cc:2258] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:23.885113 3764 raft_consensus.cc:2229] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:23.886251 3764 raft_consensus.cc:2258] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:23.888285 3765 ts_tablet_manager.cc:1905] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:23.888567 3764 ts_tablet_manager.cc:1905] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:23.896944 3764 ts_tablet_manager.cc:1918] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:23.897140 3764 log.cc:1196] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:23.897886 3764 ts_tablet_manager.cc:1939] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 55a0811ce05248d58240229f70f85888: Deleting consensus metadata
I20260927 05:02:23.898294 3765 ts_tablet_manager.cc:1918] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:23.898577 3765 log.cc:1196] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:23.899248 3765 ts_tablet_manager.cc:1939] T 82ec7a3a7bc84a8fb19773a4b141dd66 P 5b5eba98ea7b437ea9b53fe8c36fa7ad: Deleting consensus metadata
I20260927 05:02:23.900377 3257 catalog_manager.cc:4841] TS 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): tablet 82ec7a3a7bc84a8fb19773a4b141dd66 (table test-table [id=32a7021903114e109df2b49ca780253e]) successfully deleted
I20260927 05:02:23.900785 3257 catalog_manager.cc:4841] TS 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): tablet 82ec7a3a7bc84a8fb19773a4b141dd66 (table test-table [id=32a7021903114e109df2b49ca780253e]) successfully deleted
I20260927 05:02:23.883962 3766 raft_consensus.cc:2229] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:02:23.994710 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 3309
I20260927 05:02:24.014565 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 3463
W20260927 05:02:24.014688 3257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:34393 (error 108)
I20260927 05:02:24.016582 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.016158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52594)
0927 05:02:24.016332 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.016439 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 05:02:24.016856 3257 catalog_manager.cc:4584] TS f2c771a8f3fb41b990dc5c5ca2a96c46 (127.20.169.65:34393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82ec7a3a7bc84a8fb19773a4b141dd66: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
I20260927 05:02:24.033643 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.015461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.20.169.66:40985)
0927 05:02:24.015642 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:24.015647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:24.015757 (+ 110us) server_negotiation.cc:407] Connection header received
0927 05:02:24.015973 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:24.015979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:24.016064 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:24.016173 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:24.017663 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:24.018370 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:24.019698 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:24.020072 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:24.020198 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:24.026699 (+ 6501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:24.026715 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:24.026731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:24.026787 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:24.030317 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:24.031107 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:24.031116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:24.031119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:24.031281 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:24.031735 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:24.031755 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:24.031764 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:24.032179 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:24.032533 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:24.032965 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 05:02:24.033219 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:02:24.034363 3587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:24.070477 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.069974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52610)
0927 05:02:24.070188 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.070309 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 05:02:24.071683 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.071284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52620)
0927 05:02:24.071479 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.071573 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 05:02:24.132699 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.132135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52628)
0927 05:02:24.132386 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.132497 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:02:24.132943 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 3463
I20260927 05:02:24.133926 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.133463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52640)
0927 05:02:24.133685 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.133772 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 05:02:24.151288 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 3595
I20260927 05:02:24.165468 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.152143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.20.169.67:43007)
0927 05:02:24.152324 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:24.152330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:24.152399 (+ 69us) server_negotiation.cc:407] Connection header received
0927 05:02:24.152602 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:24.152608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:24.152693 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:24.152811 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:24.154155 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:24.154887 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:24.156230 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:24.156623 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:24.156776 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:24.159405 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:24.159427 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:24.159445 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:24.159477 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:24.162472 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:24.163232 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:24.163240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:24.163242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:24.163387 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:24.163798 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:24.163808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:24.163811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:24.164186 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:24.164492 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:24.164861 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 05:02:24.165079 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:02:24.166205 3719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:24.214591 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.213978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52642)
0927 05:02:24.214245 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.214388 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 05:02:24.215888 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.215376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52648)
0927 05:02:24.215649 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.215741 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 05:02:24.257409 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 3595
I20260927 05:02:24.275000 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 3239
I20260927 05:02:24.276134 3302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:24.390749 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:24.389798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34393 (local address 127.0.0.1:52656)
0927 05:02:24.389988 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:24.390160 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:02:24.390803 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 3239
I20260927 05:02:24.413856 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38855
--webserver_interface=127.20.169.126
--webserver_port=42555
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:24.756765 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:24.757121 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:24.757284 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:24.764411 3773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:24.764509 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:24.764588 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:24.764690 3773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:24.764750 3773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:24.775019 3773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:24.776330 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:24.777719 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:24.789556 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:24.789620 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:24.789759 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:24.790508 3773 server_base.cc:1034] running on GCE node
I20260927 05:02:24.791352 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:24.792773 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:24.794049 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790485344793994 us; error 44 us; skew 500 ppm
I20260927 05:02:24.794438 3773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:24 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485344, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:24.797574 3773 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:24.799005 3773 webserver.cc:460] Webserver started at http://127.20.169.126:42555/ using document root <none> and password file <none>
I20260927 05:02:24.799667 3773 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:24.799808 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:24.805166 3773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:24.807921 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:24.809409 3773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:24.809592 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "791da222f14b4b6b93cb3eb7995e4d99"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "7059b3e51b12a722a87a1bda9309f699"
server_key_iv: "c82fcafe66ce3b569a6121d4b78a27fa"
server_key_version: "encryptionkey@0"
I20260927 05:02:24.810568 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:24.825585 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:24.829541 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:24.829798 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:24.849624 3773 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:38855
I20260927 05:02:24.849632 3839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:38855 every 8 connection(s)
I20260927 05:02:24.851701 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:24.857255 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3773
I20260927 05:02:24.858196 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:34393
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:38855
--webserver_port=38247
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:24.858592 3840 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:02:24.861775 3840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap starting.
I20260927 05:02:24.881517 3840 log.cc:824] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:24.898140 3840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:02:24.898671 3840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99: Bootstrap complete.
I20260927 05:02:24.907740 3840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:24.908628 3840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791da222f14b4b6b93cb3eb7995e4d99, State: Initialized, Role: FOLLOWER
I20260927 05:02:24.909250 3840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:24.909489 3840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:24.909614 3840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:24.909829 3840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:02:24.913738 3840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:24.914302 3840 leader_election.cc:302] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791da222f14b4b6b93cb3eb7995e4d99; no voters:
I20260927 05:02:24.915000 3840 leader_election.cc:288] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 05:02:24.915133 3844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:02:24.915899 3844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [term 3 LEADER]: Becoming Leader. State: Replica: 791da222f14b4b6b93cb3eb7995e4d99, State: Running, Role: LEADER
I20260927 05:02:24.916633 3844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } }
I20260927 05:02:24.918517 3844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791da222f14b4b6b93cb3eb7995e4d99. Latest consensus state: current_term: 3 leader_uuid: "791da222f14b4b6b93cb3eb7995e4d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } } }
I20260927 05:02:24.918959 3844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:24.919252 3846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "791da222f14b4b6b93cb3eb7995e4d99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791da222f14b4b6b93cb3eb7995e4d99" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 38855 } } }
I20260927 05:02:24.919581 3846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:24.923600 3840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791da222f14b4b6b93cb3eb7995e4d99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:24.927485 3848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:24.937777 3848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:24.939658 3848 catalog_manager.cc:1259] Loaded cluster ID: 37d6ec98c74d46e28d642ec8ec6b5826
I20260927 05:02:24.939903 3848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 05:02:24.942463 3848 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fef333c09ad google::LogMessage::Fail() at ??:0
@ 0x7fef333c388a google::LogMessage::SendToLog() at ??:0
@ 0x7fef333c04de google::LogMessage::Flush() at ??:0
@ 0x7fef333c1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fef48df3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fef48df0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fef48e78610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fef48e78439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fef3600c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fef3600f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fef3600f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fef35feaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef392c46db start_thread at ??:0
@ 0x7fef3225a61f clone at ??:0
W20260927 05:02:25.224264 3842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:25.224603 3842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:25.224794 3842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:25.233825 3842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:25.233982 3842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:25.234170 3842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:02:25.254971 3842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:34393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=38247
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:25.256755 3842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:25.258566 3842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:25.272302 3881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:25.272310 3884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:25.272193 3882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:25.273587 3842 server_base.cc:1034] running on GCE node
I20260927 05:02:25.274139 3842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:25.274999 3842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:25.276197 3842 hybrid_clock.cc:648] HybridClock initialized: now 1790485345276157 us; error 26 us; skew 500 ppm
I20260927 05:02:25.276471 3842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:25 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:25.279806 3842 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:02:25.281343 3842 webserver.cc:460] Webserver started at http://127.20.169.65:38247/ using document root <none> and password file <none>
I20260927 05:02:25.282028 3842 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:25.282155 3842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:25.287719 3842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:02:25.290624 3891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:25.291983 3842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:02:25.292165 3842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46"
format_stamp: "Formatted at 2026-09-27 05:02:19 on dist-test-slave-39f2"
server_key: "03cb9605b2a41b2d6a0ccd4d3c556e11"
server_key_iv: "f2aee4fdfb39e7c7f30c708d51564baa"
server_key_version: "encryptionkey@0"
I20260927 05:02:25.293159 3842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:25.310882 3842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:25.315197 3842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:25.315518 3842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:25.316521 3842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:25.318358 3898 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 05:02:25.324072 3842 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 05:02:25.324179 3842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 05:02:25.324364 3842 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 05:02:25.327665 3842 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 05:02:25.327742 3842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260927 05:02:25.328102 3898 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Bootstrap starting.
I20260927 05:02:25.355216 3898 log.cc:824] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:25.369863 3898 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:02:25.370596 3898 tablet_bootstrap.cc:492] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Bootstrap complete.
I20260927 05:02:25.371356 3842 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:34393
I20260927 05:02:25.371362 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:34393 every 8 connection(s)
I20260927 05:02:25.372051 3898 ts_tablet_manager.cc:1397] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Time spent bootstrapping tablet: real 0.044s user 0.028s sys 0.012s
I20260927 05:02:25.372658 3842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:02:25.374011 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 3842
I20260927 05:02:25.374766 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:40985
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:38855
--webserver_port=37133
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:02:25.381165 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.378803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:46111)
0927 05:02:25.379737 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.380126 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":416,"threads_started":1}
W20260927 05:02:25.381974 4007 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:38855 (0 consecutive failures): Network error: Failed to ping master at 127.20.169.126:38855: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
I20260927 05:02:25.382826 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.382486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:53787)
0927 05:02:25.382636 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.382727 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:02:25.384194 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.383536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:39817)
0927 05:02:25.383988 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.384074 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
W20260927 05:02:25.384411 4007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:02:25.383872 3898 raft_consensus.cc:348] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:25.385334 3898 raft_consensus.cc:729] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c771a8f3fb41b990dc5c5ca2a96c46, State: Initialized, Role: FOLLOWER
I20260927 05:02:25.385953 3898 consensus_queue.cc:260] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:25.387542 3898 ts_tablet_manager.cc:1428] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260927 05:02:25.387627 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.387211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:49899)
0927 05:02:25.387388 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.387472 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:02:25.388474 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.388166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:43869)
0927 05:02:25.388314 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.388373 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:02:25.707105 4010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:25.707370 4010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:25.707547 4010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:25.714689 4010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:25.714785 4010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:25.714977 4010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:02:25.724720 4010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:40985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=37133
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:25.726049 4010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:25.727420 4010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:25.740693 4016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:25.740752 4017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:25.740763 4019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:25.741623 4010 server_base.cc:1034] running on GCE node
I20260927 05:02:25.742205 4010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:25.743129 4010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:25.744329 4010 hybrid_clock.cc:648] HybridClock initialized: now 1790485345744286 us; error 40 us; skew 500 ppm
I20260927 05:02:25.744645 4010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:25 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:25.747740 4010 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:02:25.749266 4010 webserver.cc:460] Webserver started at http://127.20.169.66:37133/ using document root <none> and password file <none>
I20260927 05:02:25.749924 4010 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:25.750063 4010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:25.755306 4010 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:02:25.758030 4026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:25.759251 4010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:02:25.759423 4010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "55a0811ce05248d58240229f70f85888"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "fbc86294e5c4472dd918a58d21af1125"
server_key_iv: "1a4db13874555228e312bc6df144e17d"
server_key_version: "encryptionkey@0"
I20260927 05:02:25.760380 4010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:25.775015 4010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:25.778811 4010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:25.779095 4010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:25.780049 4010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:25.781430 4010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:25.781556 4010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:25.781689 4010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:25.781777 4010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:25.815889 4010 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:40985
I20260927 05:02:25.815974 4139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:40985 every 8 connection(s)
I20260927 05:02:25.817324 4010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:02:25.822124 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4010
I20260927 05:02:25.822666 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.820716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:56969)
0927 05:02:25.821525 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.821745 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":270,"threads_started":1}
I20260927 05:02:25.822747 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:43007
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:38855
--webserver_port=37281
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:33027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:25.823532 4140 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:38855 (0 consecutive failures): Network error: Failed to ping master at 127.20.169.126:38855: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
I20260927 05:02:25.824463 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.824062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:40479)
0927 05:02:25.824207 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.824273 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 05:02:25.825387 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:25.825039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:47245)
0927 05:02:25.825177 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:25.825249 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 05:02:25.825640 4140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:02:26.157459 4143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:26.157738 4143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:26.157887 4143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:26.166159 4143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:26.166263 4143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:26.166437 4143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:02:26.176255 4143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:33027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:43007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=37281
--tserver_master_addrs=127.20.169.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:26.177636 4143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:26.178923 4143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:26.192135 4148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:26.192098 4149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:26.192353 4151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:26.193416 4143 server_base.cc:1034] running on GCE node
I20260927 05:02:26.193984 4143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:26.194878 4143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:26.196106 4143 hybrid_clock.cc:648] HybridClock initialized: now 1790485346196026 us; error 51 us; skew 500 ppm
I20260927 05:02:26.196485 4143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:26 dist-test-slave-39f2 krb5kdc[2684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485346, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:26.199791 4143 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:02:26.201213 4143 webserver.cc:460] Webserver started at http://127.20.169.67:37281/ using document root <none> and password file <none>
I20260927 05:02:26.201879 4143 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:26.202009 4143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:26.207580 4143 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:02:26.210256 4158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:26.211630 4143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:02:26.211817 4143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad"
format_stamp: "Formatted at 2026-09-27 05:02:20 on dist-test-slave-39f2"
server_key: "1a55edd7faf4ca4a288d157c46b17fa1"
server_key_iv: "afd216f71e64a35e03158110b3fce450"
server_key_version: "encryptionkey@0"
I20260927 05:02:26.212771 4143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:26.233181 4143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:26.237792 4143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:26.238118 4143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:26.239259 4143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:26.240922 4143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:26.241056 4143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:26.241227 4143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:26.241341 4143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:26.279891 4143 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:43007
I20260927 05:02:26.279985 4271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:43007 every 8 connection(s)
I20260927 05:02:26.281687 4143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:02:26.286917 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.284926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:32983)
0927 05:02:26.285698 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.285968 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":293,"threads_started":1}
W20260927 05:02:26.287662 4272 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:38855 (0 consecutive failures): Network error: Failed to ping master at 127.20.169.126:38855: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
I20260927 05:02:26.288130 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4143
I20260927 05:02:26.288473 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.288139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:51293)
0927 05:02:26.288286 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.288372 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:02:26.289405 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.289072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:54907)
0927 05:02:26.289214 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.289296 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:02:26.289712 4272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:02:26.390249 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.389408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:40075)
0927 05:02:26.389941 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.390043 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":246,"threads_started":1}
I20260927 05:02:26.827056 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.826195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:36337)
0927 05:02:26.826658 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.826821 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":217,"threads_started":1}
I20260927 05:02:26.898079 4278 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:26.898434 4278 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:26.899914 4278 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
Sep 27 05:02:26 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
Sep 27 05:02:26 dist-test-slave-39f2 krb5kdc[2684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:02:26.921485 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.901021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:43007 (local address 127.20.169.65:42355)
0927 05:02:26.901712 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.901729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:26.901870 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:26.904152 (+ 2282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:26.904163 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:26.905209 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:26.905898 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:26.905920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.907435 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.907439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:26.908949 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:26.908966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.910107 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.910114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:26.910450 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:26.911102 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:26.911123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:26.914368 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:26.917960 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:26.917970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:26.917984 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:26.918479 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:26.918920 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:26.918925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:26.918929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:26.919140 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:26.920068 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:26.920072 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:26.920421 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 05:02:26.920741 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 05:02:26.921140 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":224,"spinlock_wait_cycles":28800,"thread_start_us":348,"threads_started":1}
I20260927 05:02:26.921813 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.900249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:40985 (local address 127.20.169.65:49533)
0927 05:02:26.900838 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:26.900905 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:26.901525 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:26.904153 (+ 2628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:26.904162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:26.905208 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:26.905925 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:26.905936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.907440 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.907444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:26.908936 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:26.908948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.910123 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.910130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:26.910448 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:26.911293 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:26.911317 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:26.914367 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:26.918122 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:26.918128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:26.918132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:26.918561 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:26.918976 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:26.918982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:26.918989 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:26.919147 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:26.919904 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:26.920263 (+ 359us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:26.920415 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 05:02:26.920755 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 05:02:26.921324 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":187,"spinlock_wait_cycles":9600,"thread_start_us":302,"threads_started":1}
I20260927 05:02:26.921846 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.900471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:49533 (local address 127.20.169.66:40985)
0927 05:02:26.901120 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:26.901137 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:26.901190 (+ 53us) server_negotiation.cc:407] Connection header received
0927 05:02:26.903407 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:26.903415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:26.903745 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:26.904067 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:26.906120 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.907249 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.909075 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.909884 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.910109 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:26.914664 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:26.914671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:26.914683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:26.914718 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:26.917992 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:26.918707 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:26.918713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:26.918719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:26.918860 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:26.919262 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:26.919274 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:26.919277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:26.919761 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:26.919963 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:26.921040 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0927 05:02:26.921461 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260927 05:02:26.923056 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:26.901405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:42355 (local address 127.20.169.67:43007)
0927 05:02:26.901919 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:26.901935 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:26.901968 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:02:26.903327 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:26.903336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:26.903715 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:26.904012 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:26.906091 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.907236 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.909124 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:26.909880 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:26.910090 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:26.914652 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:26.914659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:26.914673 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:26.914707 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:26.917810 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:26.918626 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:26.918634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:26.918640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:26.918796 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:26.919260 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:26.919265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:26.919274 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:26.919847 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:26.920015 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:26.922277 (+ 2262us) server_negotiation.cc:299] Negotiation successful
0927 05:02:26.922686 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
W20260927 05:02:26.925599 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
W20260927 05:02:26.926020 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:26.926156 3893 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2c771a8f3fb41b990dc5c5ca2a96c46; no voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad
I20260927 05:02:26.926520 4278 raft_consensus.cc:2726] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:02:27.291184 4283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:27.290335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:44111)
0927 05:02:27.290803 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:27.290952 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":226,"threads_started":1}
I20260927 05:02:27.391646 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:27.391173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:48973)
0927 05:02:27.391398 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:27.391495 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:02:27.828574 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:27.827869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:49973)
0927 05:02:27.828322 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:27.828417 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260927 05:02:28.292925 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:28.292143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:59171)
0927 05:02:28.292603 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:28.292679 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":273,"threads_started":1}
I20260927 05:02:28.393509 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:28.392622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:37897)
0927 05:02:28.393184 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:28.393326 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
I20260927 05:02:28.597829 4287 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:28.598048 4287 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:28.598768 4287 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
W20260927 05:02:28.599756 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
W20260927 05:02:28.599951 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:28.600028 3893 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2c771a8f3fb41b990dc5c5ca2a96c46; no voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad
I20260927 05:02:28.600239 4287 raft_consensus.cc:2726] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:02:28.830264 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:28.829580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:51031)
0927 05:02:28.830031 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:28.830108 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":247,"threads_started":1}
I20260927 05:02:29.294603 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:29.293825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:44705)
0927 05:02:29.294336 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:29.294441 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260927 05:02:29.395079 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:29.394351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:55621)
0927 05:02:29.394839 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:29.394924 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":261,"threads_started":1}
I20260927 05:02:29.831854 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:29.831127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:46237)
0927 05:02:29.831592 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:29.831677 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260927 05:02:30.296331 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:30.295495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:57833)
0927 05:02:30.296046 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:30.296130 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
I20260927 05:02:30.397353 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:30.396234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:39973)
0927 05:02:30.396974 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:30.397099 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":392,"threads_started":1}
I20260927 05:02:30.833673 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:30.832957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:52437)
0927 05:02:30.833426 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:30.833507 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":218,"threads_started":1}
I20260927 05:02:31.298260 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:31.297483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:49187)
0927 05:02:31.298011 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:31.298103 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260927 05:02:31.398836 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:31.398183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:53351)
0927 05:02:31.398580 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:31.398666 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":189,"threads_started":1}
I20260927 05:02:31.641399 4297 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:31.641582 4297 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:31.642374 4297 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
W20260927 05:02:31.643427 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
W20260927 05:02:31.643621 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:31.643714 3893 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2c771a8f3fb41b990dc5c5ca2a96c46; no voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad
I20260927 05:02:31.643961 4297 raft_consensus.cc:2726] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:02:31.835398 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:31.834601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:57471)
0927 05:02:31.835068 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:31.835162 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260927 05:02:32.299798 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:32.299115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:43335)
0927 05:02:32.299554 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:32.299634 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260927 05:02:32.400465 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:32.399725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:38299)
0927 05:02:32.400150 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:32.400228 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":229,"threads_started":1}
I20260927 05:02:32.838040 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:32.836690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:36361)
0927 05:02:32.837643 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:32.837784 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":482,"threads_started":1}
I20260927 05:02:33.302050 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:33.300926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:50623)
0927 05:02:33.301582 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:33.301696 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":246,"threads_started":1}
I20260927 05:02:33.402359 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:33.401379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:43495)
0927 05:02:33.402004 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:33.402122 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":384,"threads_started":1}
I20260927 05:02:33.435432 4304 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:33.435652 4304 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:33.436503 4304 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
W20260927 05:02:33.437822 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
W20260927 05:02:33.439693 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:33.439800 3893 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2c771a8f3fb41b990dc5c5ca2a96c46; no voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad
I20260927 05:02:33.440090 4304 raft_consensus.cc:2726] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:02:33.839773 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:33.839008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:54657)
0927 05:02:33.839505 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:33.839592 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":283,"threads_started":1}
I20260927 05:02:34.303750 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:34.303062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:40159)
0927 05:02:34.303480 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:34.303563 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260927 05:02:34.404000 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:34.403260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:34089)
0927 05:02:34.403744 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:34.403835 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260927 05:02:34.841677 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:34.840946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:43241)
0927 05:02:34.841433 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:34.841519 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
I20260927 05:02:35.305523 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:35.304687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:60589)
0927 05:02:35.305276 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:35.305364 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":335,"threads_started":1}
I20260927 05:02:35.405741 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:35.405065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:50983)
0927 05:02:35.405497 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:35.405584 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260927 05:02:35.423079 4311 raft_consensus.cc:482] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:35.423417 4311 raft_consensus.cc:504] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a0811ce05248d58240229f70f85888" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 40985 } } peers { permanent_uuid: "5b5eba98ea7b437ea9b53fe8c36fa7ad" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43007 } } peers { permanent_uuid: "f2c771a8f3fb41b990dc5c5ca2a96c46" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34393 } }
I20260927 05:02:35.424095 4311 leader_election.cc:288] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985), 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007)
W20260927 05:02:35.425163 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55a0811ce05248d58240229f70f85888 (127.20.169.66:40985): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
W20260927 05:02:35.425379 3893 leader_election.cc:341] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b5eba98ea7b437ea9b53fe8c36fa7ad (127.20.169.67:43007): Not found: Tablet not found: 82ec7a3a7bc84a8fb19773a4b141dd66
I20260927 05:02:35.425475 3893 leader_election.cc:302] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2c771a8f3fb41b990dc5c5ca2a96c46; no voters: 55a0811ce05248d58240229f70f85888, 5b5eba98ea7b437ea9b53fe8c36fa7ad
I20260927 05:02:35.425696 4311 raft_consensus.cc:2726] T 82ec7a3a7bc84a8fb19773a4b141dd66 P f2c771a8f3fb41b990dc5c5ca2a96c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:02:35.843403 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:35.842665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:48343)
0927 05:02:35.843093 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:35.843174 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":242,"threads_started":1}
I20260927 05:02:36.307315 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:36.306618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:41591)
0927 05:02:36.307060 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:36.307139 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":238,"threads_started":1}
I20260927 05:02:36.407418 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:36.406673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:59925)
0927 05:02:36.407107 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:36.407193 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
I20260927 05:02:36.845189 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:36.844373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:47751)
0927 05:02:36.844923 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:36.845009 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260927 05:02:37.309616 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:37.308493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:35439)
0927 05:02:37.309274 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:37.309425 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":450,"threads_started":1}
I20260927 05:02:37.409204 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:37.408448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:57691)
0927 05:02:37.408933 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:37.409033 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260927 05:02:37.847098 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:37.846340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:35361)
0927 05:02:37.846825 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:37.846929 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260927 05:02:38.311233 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:38.310577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:47059)
0927 05:02:38.310992 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:38.311069 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260927 05:02:38.411063 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:38.410184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:59577)
0927 05:02:38.410700 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:38.410784 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":266,"threads_started":1}
I20260927 05:02:38.848839 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:38.848023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:43745)
0927 05:02:38.848522 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:38.848635 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":262,"threads_started":1}
I20260927 05:02:39.312952 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:39.312146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:46459)
0927 05:02:39.312595 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:39.312697 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260927 05:02:39.412788 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:39.412059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:39631)
0927 05:02:39.412500 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:39.412584 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260927 05:02:39.850600 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:39.849761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:46259)
0927 05:02:39.850308 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:39.850401 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":285,"threads_started":1}
I20260927 05:02:40.314535 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:40.313832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:41477)
0927 05:02:40.314264 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:40.314345 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":228,"threads_started":1}
I20260927 05:02:40.414754 4326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:40.414014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.65:52743)
0927 05:02:40.414483 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:40.414583 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260927 05:02:40.852653 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:40.851882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.66:48159)
0927 05:02:40.852378 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:40.852477 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260927 05:02:41.295351 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 3842
I20260927 05:02:41.313485 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:41.296590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.20.169.65:34393)
0927 05:02:41.297314 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:41.297346 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:41.297776 (+ 430us) server_negotiation.cc:407] Connection header received
0927 05:02:41.298106 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:41.298118 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:41.298435 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:41.298633 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:41.300304 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:41.301859 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:41.303290 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:41.304009 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:41.304189 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:41.307164 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:41.307211 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:41.307225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:41.307267 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:41.310174 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:41.310956 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:41.310963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:41.310966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:41.311102 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:41.311505 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:41.311511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:41.311515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:41.312052 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:41.312481 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:41.312800 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 05:02:41.313149 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":352,"threads_started":1}
I20260927 05:02:41.315408 4002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:41.316733 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:41.315815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:38855 (local address 127.20.169.67:56615)
0927 05:02:41.316364 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:41.316492 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":279,"threads_started":1}
I20260927 05:02:41.413049 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 3842
I20260927 05:02:41.434122 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4010
I20260927 05:02:41.448120 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:41.434909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.20.169.66:40985)
0927 05:02:41.435447 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:41.435451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:41.435467 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:02:41.435522 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:41.435525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:41.435584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:41.435720 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:41.437231 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:41.438026 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:41.439273 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:41.439725 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:41.439931 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:41.442561 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:41.442566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:41.442569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:41.442603 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:41.445083 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:41.445745 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:41.445752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:41.445756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:41.445909 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:41.446284 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:41.446289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:41.446292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:41.446580 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:41.446870 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:41.447254 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 05:02:41.447582 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
I20260927 05:02:41.449010 4135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:41.537173 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4010
I20260927 05:02:41.557052 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4143
I20260927 05:02:41.571394 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:41.557950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.20.169.67:43007)
0927 05:02:41.558535 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:41.558539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:41.558556 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:02:41.558608 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:41.558611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:41.558676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:41.558799 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:41.560125 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:41.560962 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:41.562178 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:41.562623 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:41.562761 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:41.565351 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:41.565356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:41.565359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:41.565391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:41.568313 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:41.569167 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:41.569174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:41.569177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:41.569333 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:41.569752 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:41.569764 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:41.569766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:41.570075 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:41.570440 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:41.570700 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 05:02:41.570947 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":275,"threads_started":1}
I20260927 05:02:41.572160 4267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:41.658054 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4143
2026-09-27T05:02:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24376 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:02:41 dist-test-slave-39f2 krb5kdc[4337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:02:41 dist-test-slave-39f2 krb5kdc[4337](info): set up 2 sockets
Sep 27 05:02:41 dist-test-slave-39f2 krb5kdc[4337](info): commencing operation
krb5kdc: starting...
W20260927 05:02:42.765214 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:02:42 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:02:43.013437 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35719
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:43.406129 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:43.406426 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:43.406579 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:43.413832 4350 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:02:43.413930 4350 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:02:43.414000 4350 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:02:43.414081 4350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:02:43.414183 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:43.414310 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:43.414390 4350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:02:43.414474 4350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:02:43.424474 4350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:43.426038 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:43.427661 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:43.443373 4358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:43.443537 4356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:43.443516 4355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:43.443787 4350 server_base.cc:1034] running on GCE node
I20260927 05:02:43.445495 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:43.447062 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:43.448386 4350 hybrid_clock.cc:648] HybridClock initialized: now 1790485363448234 us; error 74 us; skew 500 ppm
I20260927 05:02:43.448764 4350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:43 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485363, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:43.455574 4350 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:02:43.457341 4350 webserver.cc:460] Webserver started at http://127.20.169.126:43097/ using document root <none> and password file <none>
I20260927 05:02:43.458113 4350 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:43.458240 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:43.458588 4350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:43.461892 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "4554ab0503bd4ac781a5a08cc424688d"
format_stamp: "Formatted at 2026-09-27 05:02:43 on dist-test-slave-39f2"
server_key: "919b481ccccc877f63d0d3df40692d55"
server_key_iv: "1c65fadb430258f2ba98e90188e7b4f7"
server_key_version: "encryptionkey@0"
I20260927 05:02:43.462702 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "4554ab0503bd4ac781a5a08cc424688d"
format_stamp: "Formatted at 2026-09-27 05:02:43 on dist-test-slave-39f2"
server_key: "919b481ccccc877f63d0d3df40692d55"
server_key_iv: "1c65fadb430258f2ba98e90188e7b4f7"
server_key_version: "encryptionkey@0"
I20260927 05:02:43.468371 4350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:02:43.472288 4365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:43.473815 4350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:43.474009 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "4554ab0503bd4ac781a5a08cc424688d"
format_stamp: "Formatted at 2026-09-27 05:02:43 on dist-test-slave-39f2"
server_key: "919b481ccccc877f63d0d3df40692d55"
server_key_iv: "1c65fadb430258f2ba98e90188e7b4f7"
server_key_version: "encryptionkey@0"
I20260927 05:02:43.474236 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:43.518006 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:43.522821 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:43.523111 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:43.542559 4350 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35719
I20260927 05:02:43.542578 4416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35719 every 8 connection(s)
I20260927 05:02:43.544050 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:02:43.548308 4417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:43.549572 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4350
I20260927 05:02:43.549851 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:02:43.550809 21157 external_mini_cluster.cc:1428] Setting key bbb16236e6e6ad5549faf9f56a43077f
I20260927 05:02:43.564131 4417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap starting.
Sep 27 05:02:43 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:43.567818 4417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:43.569000 4417 log.cc:824] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:43.571866 4417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: No bootstrap required, opened a new log
I20260927 05:02:43.574783 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:43.553244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.20.169.126:35719)
0927 05:02:43.554091 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:43.554103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:43.554144 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:02:43.555715 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:43.555737 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:43.556123 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:43.556457 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:43.558386 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:43.559717 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:43.561637 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:43.562363 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:43.562494 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:43.567289 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:43.567298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:43.567312 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:43.567356 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:43.570584 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:43.571317 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:43.571323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:43.571330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:43.571494 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:43.571901 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:43.571905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:43.571907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:43.572487 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:43.572821 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:43.573333 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 05:02:43.573763 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":386,"threads_started":1}
I20260927 05:02:43.580459 4417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:02:43.580888 4417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:43.580999 4417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4554ab0503bd4ac781a5a08cc424688d, State: Initialized, Role: FOLLOWER
I20260927 05:02:43.581705 4417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:02:43.582034 4417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:02:43.582176 4417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:02:43.582383 4417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:43.584496 4417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:02:43.585072 4417 leader_election.cc:302] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4554ab0503bd4ac781a5a08cc424688d; no voters:
I20260927 05:02:43.585667 4417 leader_election.cc:288] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:02:43.585806 4422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:43.586526 4422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 LEADER]: Becoming Leader. State: Replica: 4554ab0503bd4ac781a5a08cc424688d, State: Running, Role: LEADER
I20260927 05:02:43.587036 4422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:02:43.587819 4417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:02:43.590104 4423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4554ab0503bd4ac781a5a08cc424688d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } } }
I20260927 05:02:43.590106 4424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4554ab0503bd4ac781a5a08cc424688d. Latest consensus state: current_term: 1 leader_uuid: "4554ab0503bd4ac781a5a08cc424688d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } } }
I20260927 05:02:43.590755 4423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:43.590912 4424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: This master's current role is: LEADER
I20260927 05:02:43.594106 4427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:02:43.599145 4427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:02:43.608750 4427 catalog_manager.cc:1347] Generated new cluster ID: a4ed0ffd958e49f385a82832602e6c05
I20260927 05:02:43.608887 4427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:02:43.618948 4427 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:02:43.620749 4427 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:02:43.641965 4427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Generated new TSK 0
I20260927 05:02:43.642879 4427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:02:43.704519 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35719
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:44.039943 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:44.040216 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:44.040315 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:44.047465 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:44.047565 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:44.047674 4443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:02:44.057812 4443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:44.059088 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:44.060374 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:44.073827 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:44.073938 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:44.073944 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:44.074713 4443 server_base.cc:1034] running on GCE node
I20260927 05:02:44.075281 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:44.076148 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:44.077392 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790485364077338 us; error 37 us; skew 500 ppm
I20260927 05:02:44.077744 4443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:44 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485364, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:44.081048 4443 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:02:44.082540 4443 webserver.cc:460] Webserver started at http://127.20.169.65:44541/ using document root <none> and password file <none>
I20260927 05:02:44.083170 4443 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:44.083268 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:44.083532 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:44.086403 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "d90cb00af7d2481bb25845a3ea1ba5b7"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "2557120bfb4fe81635c814600f10cb2e"
server_key_iv: "c0a7ef69bb78349a9d85bc2fda70c784"
server_key_version: "encryptionkey@0"
I20260927 05:02:44.087072 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "d90cb00af7d2481bb25845a3ea1ba5b7"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "2557120bfb4fe81635c814600f10cb2e"
server_key_iv: "c0a7ef69bb78349a9d85bc2fda70c784"
server_key_version: "encryptionkey@0"
I20260927 05:02:44.092398 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:02:44.096019 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:44.097452 4443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:02:44.097631 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "d90cb00af7d2481bb25845a3ea1ba5b7"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "2557120bfb4fe81635c814600f10cb2e"
server_key_iv: "c0a7ef69bb78349a9d85bc2fda70c784"
server_key_version: "encryptionkey@0"
I20260927 05:02:44.097836 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:44.116600 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:44.121409 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:44.121723 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:44.122810 4443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:44.124236 4443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:44.124354 4443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:44.124516 4443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:44.124590 4443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:44.160755 4443 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:39767
I20260927 05:02:44.160899 4570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:39767 every 8 connection(s)
I20260927 05:02:44.162117 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:02:44.169572 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4443
I20260927 05:02:44.169853 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:02:44.170583 21157 external_mini_cluster.cc:1428] Setting key 0f7d3821d165c23c1fe23e4a253ae104
Sep 27 05:02:44 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485364, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:44.183838 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:44.164826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:46311 (local address 127.20.169.126:35719)
0927 05:02:44.165321 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:44.165325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:44.166382 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 05:02:44.168309 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:44.168314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:44.168419 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:44.168549 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:44.170491 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.171340 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.173405 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.173836 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.173977 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:44.177779 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:44.177785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:44.177788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:44.177824 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:44.180611 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:44.181418 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:44.181428 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:44.181431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:44.181547 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:44.182007 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:44.182012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:44.182017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:44.182295 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:44.182465 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:44.183219 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 05:02:44.183464 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260927 05:02:44.184916 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:44.165420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:46311)
0927 05:02:44.166175 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:44.166213 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:44.167987 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:44.168885 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:44.168903 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:44.169508 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:44.170285 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:44.170308 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.171528 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.171540 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:44.173251 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:44.173266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.174105 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.174112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:44.174353 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:44.175122 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:44.175151 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:44.177517 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:44.180763 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:44.180773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:44.180792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:44.181272 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:44.181665 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:44.181670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:44.181673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:44.181881 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:44.182438 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:44.182446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:44.182821 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 05:02:44.183098 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 05:02:44.183658 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":301,"threads_started":1}
I20260927 05:02:44.186604 4571 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35719
I20260927 05:02:44.186935 4571 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:44.187878 4571 heartbeater.cc:511] Master 127.20.169.126:35719 requested a full tablet report, sending...
I20260927 05:02:44.190735 4382 ts_manager.cc:194] Registered new tserver with Master: d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65:39767)
I20260927 05:02:44.193059 4382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:46311
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:02:44.226481 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35719
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:44.564203 4577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:44.564494 4577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:44.564714 4577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:44.572948 4577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:44.573055 4577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:44.573179 4577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:02:44.584080 4577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:44.585335 4577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:44.586647 4577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:44.601001 4585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:44.601042 4583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:44.601080 4582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:44.602344 4577 server_base.cc:1034] running on GCE node
I20260927 05:02:44.602995 4577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:44.603920 4577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:44.605103 4577 hybrid_clock.cc:648] HybridClock initialized: now 1790485364605057 us; error 33 us; skew 500 ppm
I20260927 05:02:44.605424 4577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:44 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485364, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:44.608500 4577 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:02:44.610282 4577 webserver.cc:460] Webserver started at http://127.20.169.66:40757/ using document root <none> and password file <none>
I20260927 05:02:44.611030 4577 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:44.611171 4577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:44.611513 4577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:44.614872 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "347533a9ed6942bab90e02697f054521"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "ca6cd05c21c6555e554af35768f1fa04"
server_key_iv: "b7ab26dcb7aadb41a738bc7fa94306ac"
server_key_version: "encryptionkey@0"
I20260927 05:02:44.615724 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "347533a9ed6942bab90e02697f054521"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "ca6cd05c21c6555e554af35768f1fa04"
server_key_iv: "b7ab26dcb7aadb41a738bc7fa94306ac"
server_key_version: "encryptionkey@0"
I20260927 05:02:44.620878 4577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:02:44.624346 4592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:44.625785 4577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:02:44.625959 4577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "347533a9ed6942bab90e02697f054521"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "ca6cd05c21c6555e554af35768f1fa04"
server_key_iv: "b7ab26dcb7aadb41a738bc7fa94306ac"
server_key_version: "encryptionkey@0"
I20260927 05:02:44.626170 4577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:44.640538 4577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:44.644672 4577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:44.645004 4577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:44.645932 4577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:44.647219 4577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:44.647322 4577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:44.647487 4577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:44.647560 4577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:44.682000 4577 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45847
I20260927 05:02:44.682042 4704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45847 every 8 connection(s)
I20260927 05:02:44.683362 4577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:02:44.691308 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4577
I20260927 05:02:44.691507 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:02:44.692056 21157 external_mini_cluster.cc:1428] Setting key e046fa760bec7f747f60d97d42dbd02e
Sep 27 05:02:44 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485364, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:44.706547 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:44.686243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40853 (local address 127.20.169.126:35719)
0927 05:02:44.686727 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:44.686733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:44.687851 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 05:02:44.689867 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:44.689872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:44.689946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:44.690047 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:44.692168 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.693070 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.695094 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.695840 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.695983 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:44.699567 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:44.699577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:44.699585 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:44.699631 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:44.703412 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:44.704156 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:44.704165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:44.704169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:44.704309 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:44.704745 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:44.704752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:44.704755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:44.705101 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:44.705259 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:44.705934 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 05:02:44.706165 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":215,"threads_started":1}
I20260927 05:02:44.707553 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:44.686863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:40853)
0927 05:02:44.687686 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:44.687726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:44.689590 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:44.690348 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:44.690357 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:44.691079 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:44.691935 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:44.691960 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.693265 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.693270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:44.694942 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:44.694954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:44.696053 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:44.696057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:44.696251 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:44.696953 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:44.696979 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:44.699327 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:44.703600 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:44.703608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:44.703621 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:44.704025 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:44.704442 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:44.704450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:44.704453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:44.704626 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:44.705233 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:44.705241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:44.705582 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 05:02:44.705860 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 05:02:44.706397 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":258,"threads_started":1}
I20260927 05:02:44.709100 4705 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35719
I20260927 05:02:44.709425 4705 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:44.710315 4705 heartbeater.cc:511] Master 127.20.169.126:35719 requested a full tablet report, sending...
I20260927 05:02:44.711983 4382 ts_manager.cc:194] Registered new tserver with Master: 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847)
I20260927 05:02:44.713232 4382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:40853
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:02:44.748070 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35719
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:02:45.079417 4711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:02:45.079700 4711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:02:45.079870 4711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:02:45.087211 4711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:02:45.087324 4711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:02:45.087487 4711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:02:45.097734 4711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:02:45.098994 4711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:02:45.100373 4711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:02:45.114639 4717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:45.114615 4716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:02:45.114676 4719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:02:45.115559 4711 server_base.cc:1034] running on GCE node
I20260927 05:02:45.116154 4711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:02:45.117391 4711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:02:45.118583 4711 hybrid_clock.cc:648] HybridClock initialized: now 1790485365118538 us; error 45 us; skew 500 ppm
I20260927 05:02:45.118886 4711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:02:45 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485365, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:02:45.122139 4711 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:02:45.123688 4711 webserver.cc:460] Webserver started at http://127.20.169.67:39645/ using document root <none> and password file <none>
I20260927 05:02:45.124368 4711 fs_manager.cc:359] Metadata directory not provided
I20260927 05:02:45.124514 4711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:02:45.124857 4711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:02:45.127714 4711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "a41f5d12cfd7496c84daadc1404ae254"
format_stamp: "Formatted at 2026-09-27 05:02:45 on dist-test-slave-39f2"
server_key: "90703abc048903da295ec81eeda2d60e"
server_key_iv: "503c203c8a819dbe5959b9cba64ee489"
server_key_version: "encryptionkey@0"
I20260927 05:02:45.128448 4711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "a41f5d12cfd7496c84daadc1404ae254"
format_stamp: "Formatted at 2026-09-27 05:02:45 on dist-test-slave-39f2"
server_key: "90703abc048903da295ec81eeda2d60e"
server_key_iv: "503c203c8a819dbe5959b9cba64ee489"
server_key_version: "encryptionkey@0"
I20260927 05:02:45.134034 4711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 05:02:45.137651 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:45.138952 4711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:02:45.139124 4711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "a41f5d12cfd7496c84daadc1404ae254"
format_stamp: "Formatted at 2026-09-27 05:02:45 on dist-test-slave-39f2"
server_key: "90703abc048903da295ec81eeda2d60e"
server_key_iv: "503c203c8a819dbe5959b9cba64ee489"
server_key_version: "encryptionkey@0"
I20260927 05:02:45.139305 4711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:02:45.158862 4711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:02:45.163333 4711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:02:45.163619 4711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:02:45.164677 4711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:02:45.166028 4711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:02:45.166132 4711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:45.166281 4711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:02:45.166353 4711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:02:45.196365 4571 heartbeater.cc:503] Master 127.20.169.126:35719 was elected leader, sending a full tablet report...
I20260927 05:02:45.201435 4711 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:33927
I20260927 05:02:45.201562 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:33927 every 8 connection(s)
I20260927 05:02:45.202879 4711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:02:45.212414 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4711
I20260927 05:02:45.212626 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:02:45.213122 21157 external_mini_cluster.cc:1428] Setting key ba5a10962ea329f00374e234c788fc24
Sep 27 05:02:45 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485365, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:02:45.224402 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.205786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:57611 (local address 127.20.169.126:35719)
0927 05:02:45.205986 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.205991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.207088 (+ 1097us) server_negotiation.cc:407] Connection header received
0927 05:02:45.209359 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.209365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.209468 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.209593 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:45.211234 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.212083 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.214138 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.214717 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.214815 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.218969 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:45.218974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:45.218977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:45.219009 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:45.221645 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.222372 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.222376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.222378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.222490 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.222910 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.222913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.222915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.223171 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:45.223287 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.224003 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.224169 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:02:45.225521 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.206163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:57611)
0927 05:02:45.206917 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:45.206967 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:45.209062 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:45.209857 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:45.209866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:45.210439 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:02:45.211045 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.211063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.212259 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.212264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.213953 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.213969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.214961 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.214969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.215323 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.216208 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:02:45.216249 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:02:45.218731 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:02:45.221777 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:45.221784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:45.221806 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:45.222257 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:45.222586 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:02:45.222589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:02:45.222592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:02:45.222778 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:02:45.223278 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:02:45.223284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:02:45.223619 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 05:02:45.223886 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 05:02:45.224284 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":274,"threads_started":1}
I20260927 05:02:45.227037 4839 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35719
I20260927 05:02:45.227478 4839 heartbeater.cc:464] Registering TS with master...
I20260927 05:02:45.228549 4839 heartbeater.cc:511] Master 127.20.169.126:35719 requested a full tablet report, sending...
I20260927 05:02:45.230605 4382 ts_manager.cc:194] Registered new tserver with Master: a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927)
I20260927 05:02:45.231521 4382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:57611
I20260927 05:02:45.242316 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:02:45.261801 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.247833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.20.169.126:35719)
0927 05:02:45.248049 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.248054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.248541 (+ 487us) server_negotiation.cc:407] Connection header received
0927 05:02:45.248675 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.248679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.248743 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.248898 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:45.250318 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.251055 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.252415 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.252963 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.253076 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.255508 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:45.255513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:45.255517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:45.255550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:45.258615 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.259488 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.259492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.259495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.259630 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.260123 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.260128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.260130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.260506 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:45.260725 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.261291 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.261504 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 05:02:45.267072 4382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:02:45.274863 4382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:02:45.315331 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.289246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.20.169.66:45847)
0927 05:02:45.297328 (+ 8082us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.297346 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.305199 (+ 7853us) server_negotiation.cc:407] Connection header received
0927 05:02:45.305406 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.305421 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.305823 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.306075 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:45.307252 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.309043 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.312007 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.313407 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.313600 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.313850 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.314712 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.314934 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7805,"thread_start_us":7641,"threads_started":1}
I20260927 05:02:45.316000 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.289729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.20.169.65:39767)
0927 05:02:45.305865 (+ 16136us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.305876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.306424 (+ 548us) server_negotiation.cc:407] Connection header received
0927 05:02:45.306529 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.306536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.306795 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.306979 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:45.307950 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.309458 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.312555 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.314125 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.314252 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.314493 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.315372 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.315577 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15945,"thread_start_us":15745,"threads_started":1}
I20260927 05:02:45.316442 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.288395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45847 (local address 127.0.0.1:41752)
0927 05:02:45.305027 (+ 16632us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:45.305052 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:45.305203 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:45.306193 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:45.306202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:45.306239 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:45.306908 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.306930 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.309198 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.309204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.311764 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.311777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.313663 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.313671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.314009 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.314085 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 05:02:45.314245 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 05:02:45.314463 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16481,"mutex_wait_us":102,"thread_start_us":17475,"threads_started":1}
I20260927 05:02:45.316473 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.290177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:41574)
0927 05:02:45.306147 (+ 15970us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:45.306165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:45.306286 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:45.307331 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:45.307336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:45.307364 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:45.307714 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.307727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.309628 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.309636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.312368 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.312392 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.314725 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.314733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.314923 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.314957 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:02:45.315047 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 05:02:45.315177 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15859,"mutex_wait_us":37,"thread_start_us":14526,"threads_started":1}
I20260927 05:02:45.317123 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.287980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:33927 (local address 127.0.0.1:49636)
0927 05:02:45.302056 (+ 14076us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:45.302125 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:45.304346 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:45.308294 (+ 3948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:45.308301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:45.308332 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:45.308679 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.308690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.311906 (+ 3216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.311911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.313953 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.313968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.315869 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.315876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.316104 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.316150 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 05:02:45.316211 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 05:02:45.316400 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13734,"mutex_wait_us":3,"thread_start_us":14186,"threads_started":1}
I20260927 05:02:45.317447 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.288357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.20.169.67:33927)
0927 05:02:45.307622 (+ 19265us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.307634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.307665 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:02:45.307788 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.307796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.308044 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.308915 (+ 871us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:45.309259 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.311746 (+ 2487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.314145 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.315624 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.316316 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.316526 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.316918 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.317090 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19084,"thread_start_us":18941,"threads_started":1}
I20260927 05:02:45.320986 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 5502fba1dba540389249cd90a888e92c (DEFAULT_TABLE table=test-table [id=ede24c930fda4034b3bb575437f12912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:45.321087 4640 tablet_service.cc:1467] Processing CreateTablet for tablet 5502fba1dba540389249cd90a888e92c (DEFAULT_TABLE table=test-table [id=ede24c930fda4034b3bb575437f12912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:45.321282 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 5502fba1dba540389249cd90a888e92c (DEFAULT_TABLE table=test-table [id=ede24c930fda4034b3bb575437f12912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:02:45.322651 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502fba1dba540389249cd90a888e92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:45.322700 4640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502fba1dba540389249cd90a888e92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:45.322700 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502fba1dba540389249cd90a888e92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:02:45.337200 4857 tablet_bootstrap.cc:492] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Bootstrap starting.
I20260927 05:02:45.340586 4858 tablet_bootstrap.cc:492] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Bootstrap starting.
I20260927 05:02:45.341351 4857 tablet_bootstrap.cc:654] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:45.343088 4857 log.cc:824] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:45.345077 4858 tablet_bootstrap.cc:654] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:45.346431 4859 tablet_bootstrap.cc:492] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Bootstrap starting.
I20260927 05:02:45.346632 4858 log.cc:824] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:45.347715 4857 tablet_bootstrap.cc:492] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: No bootstrap required, opened a new log
I20260927 05:02:45.348067 4857 ts_tablet_manager.cc:1397] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 05:02:45.349802 4859 tablet_bootstrap.cc:654] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Neither blocks nor log segments found. Creating new log.
I20260927 05:02:45.350282 4858 tablet_bootstrap.cc:492] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: No bootstrap required, opened a new log
I20260927 05:02:45.350574 4858 ts_tablet_manager.cc:1397] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260927 05:02:45.351017 4859 log.cc:824] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Log is configured to *not* fsync() on all Append() calls
I20260927 05:02:45.353700 4859 tablet_bootstrap.cc:492] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: No bootstrap required, opened a new log
I20260927 05:02:45.353996 4859 ts_tablet_manager.cc:1397] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:02:45.360682 4857 raft_consensus.cc:348] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.361297 4857 raft_consensus.cc:374] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:45.361435 4857 raft_consensus.cc:729] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41f5d12cfd7496c84daadc1404ae254, State: Initialized, Role: FOLLOWER
I20260927 05:02:45.362217 4857 consensus_queue.cc:260] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.363554 4839 heartbeater.cc:503] Master 127.20.169.126:35719 was elected leader, sending a full tablet report...
I20260927 05:02:45.363301 4858 raft_consensus.cc:348] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.363816 4858 raft_consensus.cc:374] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:45.363966 4858 raft_consensus.cc:729] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347533a9ed6942bab90e02697f054521, State: Initialized, Role: FOLLOWER
I20260927 05:02:45.364635 4857 ts_tablet_manager.cc:1428] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260927 05:02:45.364777 4858 consensus_queue.cc:260] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.364959 4859 raft_consensus.cc:348] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.365451 4859 raft_consensus.cc:374] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:02:45.365571 4859 raft_consensus.cc:729] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90cb00af7d2481bb25845a3ea1ba5b7, State: Initialized, Role: FOLLOWER
I20260927 05:02:45.366221 4859 consensus_queue.cc:260] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.366566 4705 heartbeater.cc:503] Master 127.20.169.126:35719 was elected leader, sending a full tablet report...
I20260927 05:02:45.367324 4858 ts_tablet_manager.cc:1428] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Time spent starting tablet: real 0.017s user 0.015s sys 0.003s
I20260927 05:02:45.369021 4859 ts_tablet_manager.cc:1428] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:02:45.385648 4865 raft_consensus.cc:482] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:02:45.386041 4865 raft_consensus.cc:504] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.388358 4865 leader_election.cc:288] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927), 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847)
I20260927 05:02:45.396500 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.388672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:57859 (local address 127.20.169.67:33927)
0927 05:02:45.388937 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.388946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.390164 (+ 1218us) server_negotiation.cc:407] Connection header received
0927 05:02:45.390411 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.390417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.390492 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.390595 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:45.391297 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.392118 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.394441 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.395414 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.395598 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.395681 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.396069 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.396228 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:02:45.396616 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.388644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:33927 (local address 127.20.169.65:57859)
0927 05:02:45.390041 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:45.390061 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:45.390252 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:45.390611 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:45.390616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:45.390641 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:45.391127 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.391138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.392258 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.392267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.394268 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.394282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.395654 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.395661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.395879 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.395928 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 05:02:45.396010 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 05:02:45.396109 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":39,"spinlock_wait_cycles":21248,"thread_start_us":819,"threads_started":1}
I20260927 05:02:45.396709 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.389051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:52481 (local address 127.20.169.66:45847)
0927 05:02:45.389256 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.389260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.390380 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 05:02:45.390598 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.390602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.390664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.390770 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:02:45.391473 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.392230 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.394503 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.395560 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.395722 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.395816 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.396280 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.396421 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:02:45.397411 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.388858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45847 (local address 127.20.169.65:52481)
0927 05:02:45.390248 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0927 05:02:45.390264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:02:45.390432 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:02:45.390787 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:02:45.390792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:02:45.390839 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:02:45.391273 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.391283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.392393 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.392399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.394288 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:02:45.394309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.395873 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.395878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:02:45.396084 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.396118 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:02:45.396216 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:02:45.396360 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":29,"thread_start_us":1016,"threads_started":1}
I20260927 05:02:45.398010 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502fba1dba540389249cd90a888e92c" candidate_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41f5d12cfd7496c84daadc1404ae254" is_pre_election: true
I20260927 05:02:45.398437 4660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502fba1dba540389249cd90a888e92c" candidate_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347533a9ed6942bab90e02697f054521" is_pre_election: true
I20260927 05:02:45.398545 4794 raft_consensus.cc:2454] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90cb00af7d2481bb25845a3ea1ba5b7 in term 0.
I20260927 05:02:45.398962 4660 raft_consensus.cc:2454] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90cb00af7d2481bb25845a3ea1ba5b7 in term 0.
I20260927 05:02:45.399267 4460 leader_election.cc:302] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41f5d12cfd7496c84daadc1404ae254, d90cb00af7d2481bb25845a3ea1ba5b7; no voters:
I20260927 05:02:45.399752 4865 raft_consensus.cc:2781] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:02:45.399955 4865 raft_consensus.cc:482] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:02:45.400084 4865 raft_consensus.cc:3037] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:45.402664 4865 raft_consensus.cc:504] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
I20260927 05:02:45.403404 4865 leader_election.cc:288] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 1 election: Requested vote from peers a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927), 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847)
I20260927 05:02:45.403779 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502fba1dba540389249cd90a888e92c" candidate_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41f5d12cfd7496c84daadc1404ae254"
I20260927 05:02:45.403908 4660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502fba1dba540389249cd90a888e92c" candidate_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347533a9ed6942bab90e02697f054521"
I20260927 05:02:45.404007 4794 raft_consensus.cc:3037] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:45.404161 4660 raft_consensus.cc:3037] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:02:45.406898 4794 raft_consensus.cc:2454] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90cb00af7d2481bb25845a3ea1ba5b7 in term 1.
I20260927 05:02:45.407016 4660 raft_consensus.cc:2454] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90cb00af7d2481bb25845a3ea1ba5b7 in term 1.
I20260927 05:02:45.407337 4460 leader_election.cc:302] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41f5d12cfd7496c84daadc1404ae254, d90cb00af7d2481bb25845a3ea1ba5b7; no voters:
I20260927 05:02:45.407668 4865 raft_consensus.cc:2781] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:02:45.408145 4865 raft_consensus.cc:686] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 LEADER]: Becoming Leader. State: Replica: d90cb00af7d2481bb25845a3ea1ba5b7, State: Running, Role: LEADER
I20260927 05:02:45.408630 4865 consensus_queue.cc:237] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } }
W20260927 05:02:45.414503 4572 tablet.cc:2369] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:45.414296 4380 catalog_manager.cc:5495] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65). New cstate: current_term: 1 leader_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } health_report { overall_health: HEALTHY } } }
W20260927 05:02:45.435559 4706 tablet.cc:2369] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:45.445668 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.439824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.20.169.65:39767)
0927 05:02:45.440041 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.440047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.440066 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:02:45.440159 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.440163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.440243 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.440427 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:02:45.440869 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.441608 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.443283 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.443754 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.443850 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.444477 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:02:45.445156 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:02:45.445275 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.445407 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.445471 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:02:45.453650 4660 raft_consensus.cc:1261] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Refusing update from remote peer d90cb00af7d2481bb25845a3ea1ba5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:02:45.453655 4794 raft_consensus.cc:1261] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Refusing update from remote peer d90cb00af7d2481bb25845a3ea1ba5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 05:02:45.454576 4840 tablet.cc:2369] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:02:45.454571 4868 consensus_queue.cc:1035] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:45.454941 4865 consensus_queue.cc:1035] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:02:45.465499 4872 mvcc.cc:205] Tried to move back new op lower bound from 7333828056888754176 to 7333828056718323712. Current Snapshot: MvccSnapshot[applied={T|T < 7333828056888754176}]
I20260927 05:02:45.479928 4380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:45.480234 4380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:02:45.484153 4380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Sending DeleteTablet for 3 replicas of tablet 5502fba1dba540389249cd90a888e92c
I20260927 05:02:45.485284 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502fba1dba540389249cd90a888e92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:45 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:49636
I20260927 05:02:45.485699 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502fba1dba540389249cd90a888e92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:45 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:41574
I20260927 05:02:45.485756 4640 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502fba1dba540389249cd90a888e92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:02:45 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:41752
I20260927 05:02:45.486404 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4443
I20260927 05:02:45.502457 4882 tablet_replica.cc:330] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: stopping tablet replica
I20260927 05:02:45.503036 4882 raft_consensus.cc:2229] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:02:45.503614 4881 tablet_replica.cc:330] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: stopping tablet replica
I20260927 05:02:45.503880 4882 raft_consensus.cc:2258] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:45.504142 4881 raft_consensus.cc:2229] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:45.504745 4883 tablet_replica.cc:330] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: stopping tablet replica
I20260927 05:02:45.505200 4883 raft_consensus.cc:2229] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:02:45.505767 4883 raft_consensus.cc:2258] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:45.506480 4882 ts_tablet_manager.cc:1905] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:45.508212 4883 ts_tablet_manager.cc:1905] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:02:45 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:02:45.508723 4881 raft_consensus.cc:2258] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:02:45.516185 4881 ts_tablet_manager.cc:1905] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:02:45.516266 4883 ts_tablet_manager.cc:1918] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:45.516422 4883 log.cc:1196] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/5502fba1dba540389249cd90a888e92c
I20260927 05:02:45.517000 4883 ts_tablet_manager.cc:1939] T 5502fba1dba540389249cd90a888e92c P 347533a9ed6942bab90e02697f054521: Deleting consensus metadata
I20260927 05:02:45.518924 4369 catalog_manager.cc:4841] TS 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847): tablet 5502fba1dba540389249cd90a888e92c (table test-table [id=ede24c930fda4034b3bb575437f12912]) successfully deleted
I20260927 05:02:45.520593 4882 ts_tablet_manager.cc:1918] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:45.520778 4882 log.cc:1196] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/5502fba1dba540389249cd90a888e92c
I20260927 05:02:45.521366 4882 ts_tablet_manager.cc:1939] T 5502fba1dba540389249cd90a888e92c P d90cb00af7d2481bb25845a3ea1ba5b7: Deleting consensus metadata
I20260927 05:02:45.522804 4881 ts_tablet_manager.cc:1918] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:02:45.522987 4881 log.cc:1196] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/5502fba1dba540389249cd90a888e92c
I20260927 05:02:45.523375 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.487544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.20.169.65:39767)
0927 05:02:45.487716 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.487722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.495181 (+ 7459us) server_negotiation.cc:407] Connection header received
0927 05:02:45.496403 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.496412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.496523 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.496616 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:45.501744 (+ 5128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.503119 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.505540 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.506382 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.506560 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.512585 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:45.512602 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:45.512624 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:45.512679 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:45.517307 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.518273 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.518286 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.518290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.518463 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.518938 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.518944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.518947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.519375 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:45.519687 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.520994 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.521249 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:02:45.523315 4366 catalog_manager.cc:4841] TS d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65:39767): tablet 5502fba1dba540389249cd90a888e92c (table test-table [id=ede24c930fda4034b3bb575437f12912]) successfully deleted
I20260927 05:02:45.524029 4881 ts_tablet_manager.cc:1939] T 5502fba1dba540389249cd90a888e92c P a41f5d12cfd7496c84daadc1404ae254: Deleting consensus metadata
I20260927 05:02:45.524168 4566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:45.526290 4367 catalog_manager.cc:4841] TS a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927): tablet 5502fba1dba540389249cd90a888e92c (table test-table [id=ede24c930fda4034b3bb575437f12912]) successfully deleted
I20260927 05:02:45.627019 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4443
I20260927 05:02:45.647024 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4577
Sep 27 05:02:45 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:02:45.663365 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.647939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.20.169.66:45847)
0927 05:02:45.648111 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.648116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.648134 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:02:45.648257 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.648261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.648329 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.648412 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:45.649827 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.650610 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.652152 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.652560 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.652657 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.657634 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:45.657649 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:45.657664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:45.657703 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:45.660545 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.661365 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.661372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.661374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.661499 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.661898 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.661902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.661904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.662263 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:45.662494 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.662792 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.662990 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:02:45.664089 4700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:45.752909 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4577
I20260927 05:02:45.771001 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4711
Sep 27 05:02:45 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:02:45.786499 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:02:45.771809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.20.169.67:33927)
0927 05:02:45.771988 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:02:45.771992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:02:45.772010 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:02:45.772116 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:02:45.772121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:02:45.772181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:02:45.772302 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:02:45.773694 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.774437 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.775759 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:02:45.776145 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:02:45.776241 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:02:45.781055 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:02:45.781070 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:02:45.781087 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:02:45.781120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:02:45.783961 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.784599 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.784605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.784607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.784733 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:02:45.785120 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:02:45.785124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:02:45.785125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:02:45.785485 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:02:45.785697 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 05:02:45.785960 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 05:02:45.786120 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:02:45.787168 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:45.881716 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4711
I20260927 05:02:45.900014 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4350
I20260927 05:02:45.901348 4412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:02:46.012552 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4350
I20260927 05:03:06.035964 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35719
--webserver_interface=127.20.169.126
--webserver_port=43097
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:06.379539 4889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:06.379871 4889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:06.380048 4889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:06.388602 4889 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:03:06.388742 4889 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:03:06.388834 4889 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:03:06.388936 4889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:03:06.389005 4889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:06.389073 4889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:06.389118 4889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:03:06.389165 4889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:03:06.399148 4889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=43097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:06.400418 4889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:06.401861 4889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:06.425273 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:06.425318 4898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:06.425230 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:06.426465 4889 server_base.cc:1034] running on GCE node
I20260927 05:03:06.427368 4889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:06.429039 4889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:06.430341 4889 hybrid_clock.cc:648] HybridClock initialized: now 1790485386430238 us; error 47 us; skew 500 ppm
I20260927 05:03:06.430744 4889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:06 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485386, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:06.434487 4889 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:03:06.436206 4889 webserver.cc:460] Webserver started at http://127.20.169.126:43097/ using document root <none> and password file <none>
I20260927 05:03:06.437098 4889 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:06.437247 4889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:06.443398 4889 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 05:03:06.446825 4905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:06.448189 4889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:03:06.448431 4889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "4554ab0503bd4ac781a5a08cc424688d"
format_stamp: "Formatted at 2026-09-27 05:02:43 on dist-test-slave-39f2"
server_key: "919b481ccccc877f63d0d3df40692d55"
server_key_iv: "1c65fadb430258f2ba98e90188e7b4f7"
server_key_version: "encryptionkey@0"
I20260927 05:03:06.449515 4889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:06.470716 4889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:06.475484 4889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:06.475808 4889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:06.495990 4889 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35719
I20260927 05:03:06.496014 4956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35719 every 8 connection(s)
I20260927 05:03:06.497661 4889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:03:06.502774 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4889
I20260927 05:03:06.503819 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:39767
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:35719
--webserver_port=44541
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:03:06.505893 4957 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:03:06.511461 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap starting.
I20260927 05:03:06.532276 4957 log.cc:824] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:06.544337 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:03:06.545042 4957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap complete.
I20260927 05:03:06.554446 4957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:06.555271 4957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4554ab0503bd4ac781a5a08cc424688d, State: Initialized, Role: FOLLOWER
I20260927 05:03:06.555879 4957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:06.556123 4957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:03:06.556237 4957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:03:06.556437 4957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:03:06.560189 4957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:06.560721 4957 leader_election.cc:302] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4554ab0503bd4ac781a5a08cc424688d; no voters:
I20260927 05:03:06.561425 4957 leader_election.cc:288] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:03:06.561578 4961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:03:06.562774 4961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 LEADER]: Becoming Leader. State: Replica: 4554ab0503bd4ac781a5a08cc424688d, State: Running, Role: LEADER
I20260927 05:03:06.563412 4961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:06.563802 4957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:03:06.565971 4963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4554ab0503bd4ac781a5a08cc424688d. Latest consensus state: current_term: 2 leader_uuid: "4554ab0503bd4ac781a5a08cc424688d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } } }
I20260927 05:03:06.566435 4963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:06.567219 4962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4554ab0503bd4ac781a5a08cc424688d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } } }
I20260927 05:03:06.567595 4967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:03:06.567638 4962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:06.579150 4967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:03:06.581159 4967 catalog_manager.cc:1259] Loaded cluster ID: a4ed0ffd958e49f385a82832602e6c05
I20260927 05:03:06.581312 4967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:03:06.586824 4967 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:03:06.588877 4967 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Loaded TSK: 0
I20260927 05:03:06.646008 4979 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:03:06.653664 4967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Generated new TSK 1
I20260927 05:03:06.654150 4967 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:03:06.881705 4959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:06.882019 4959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:06.882189 4959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:06.889703 4959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:06.889873 4959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:06.890070 4959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:03:06.899964 4959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:39767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=44541
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:06.901480 4959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:06.902801 4959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:06.918026 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:06.918061 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:06.918583 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:06.919351 4959 server_base.cc:1034] running on GCE node
I20260927 05:03:06.919946 4959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:06.920979 4959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:06.922196 4959 hybrid_clock.cc:648] HybridClock initialized: now 1790485386922143 us; error 44 us; skew 500 ppm
I20260927 05:03:06.922484 4959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:06 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485386, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:06.925659 4959 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:03:06.927284 4959 webserver.cc:460] Webserver started at http://127.20.169.65:44541/ using document root <none> and password file <none>
I20260927 05:03:06.927970 4959 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:06.928092 4959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:06.933773 4959 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:03:06.936843 4995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:06.938377 4959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:03:06.938588 4959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "d90cb00af7d2481bb25845a3ea1ba5b7"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "2557120bfb4fe81635c814600f10cb2e"
server_key_iv: "c0a7ef69bb78349a9d85bc2fda70c784"
server_key_version: "encryptionkey@0"
I20260927 05:03:06.939690 4959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:06.958948 4959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:06.963661 4959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:06.963964 4959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:06.965122 4959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:06.966562 4959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:06.966701 4959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:06.966861 4959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:06.966959 4959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.003288 4959 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:39767
I20260927 05:03:07.003381 5107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:39767 every 8 connection(s)
I20260927 05:03:07.005003 4959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:03:07.012454 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 4959
I20260927 05:03:07.013324 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:45847
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:35719
--webserver_port=40757
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:03:07 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485386, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:07.032636 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.008420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:51839)
0927 05:03:07.009281 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:07.009354 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:07.011046 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:07.012515 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:07.012528 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:07.013394 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:07.014199 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:07.014230 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.015926 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.015935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:07.018120 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:07.018140 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.019184 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.019193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:07.019531 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.020345 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:07.020379 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:07.023330 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:07.028152 (+ 4822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:07.028160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:07.028173 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:07.028648 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:07.029218 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:07.029223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:07.029227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:07.029498 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:07.030537 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:07.030543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:07.030869 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 05:03:07.031104 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 05:03:07.031563 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":401,"threads_started":1}
I20260927 05:03:07.033121 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.008030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:51839 (local address 127.20.169.126:35719)
0927 05:03:07.008674 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:07.008695 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:07.009502 (+ 807us) server_negotiation.cc:407] Connection header received
0927 05:03:07.011583 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:07.011595 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:07.011966 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:07.012273 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:07.014437 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.015730 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.018300 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.018940 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.019068 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.023639 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:07.023652 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:07.023674 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:07.023725 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:07.027972 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.028843 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.028855 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.028866 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.029073 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.029629 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.029636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.029642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.030383 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:07.030585 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:07.031395 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 05:03:07.031830 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":244,"threads_started":1}
I20260927 05:03:07.036110 5108 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35719
I20260927 05:03:07.036428 5108 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:07.037230 5108 heartbeater.cc:511] Master 127.20.169.126:35719 requested a full tablet report, sending...
I20260927 05:03:07.039817 4922 ts_manager.cc:194] Registered new tserver with Master: d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65:39767)
I20260927 05:03:07.042253 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:51839
W20260927 05:03:07.349638 5112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:07.349923 5112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:07.350033 5112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:07.357220 5112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:07.357353 5112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:07.357462 5112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:03:07.367151 5112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:45847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=40757
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:07.368392 5112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:07.369668 5112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:07.382874 5118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:07.382939 5117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:07.383018 5120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:07.383677 5112 server_base.cc:1034] running on GCE node
I20260927 05:03:07.384621 5112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:07.385903 5112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:07.387089 5112 hybrid_clock.cc:648] HybridClock initialized: now 1790485387387035 us; error 46 us; skew 500 ppm
I20260927 05:03:07.387348 5112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:07 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485387, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:07.390627 5112 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:03:07.392141 5112 webserver.cc:460] Webserver started at http://127.20.169.66:40757/ using document root <none> and password file <none>
I20260927 05:03:07.392755 5112 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:07.392935 5112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:07.398123 5112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 05:03:07.400928 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.402266 5112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:03:07.402468 5112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "347533a9ed6942bab90e02697f054521"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "ca6cd05c21c6555e554af35768f1fa04"
server_key_iv: "b7ab26dcb7aadb41a738bc7fa94306ac"
server_key_version: "encryptionkey@0"
I20260927 05:03:07.403414 5112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:07.427970 5112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:07.432802 5112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:07.433158 5112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:07.434163 5112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:07.435491 5112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:07.435586 5112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.435717 5112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:07.435859 5112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.470731 5112 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45847
I20260927 05:03:07.470799 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45847 every 8 connection(s)
I20260927 05:03:07.472178 5112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:03:07.473721 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5112
I20260927 05:03:07.474359 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:33927
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:35719
--webserver_port=39645
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:03:07 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485387, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:07.503167 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.482733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:44251 (local address 127.20.169.126:35719)
0927 05:03:07.482946 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:07.482951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:07.484497 (+ 1546us) server_negotiation.cc:407] Connection header received
0927 05:03:07.486539 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:07.486544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:07.486639 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:07.486748 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:07.488344 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.489202 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.490815 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.491255 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.491386 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.495417 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:07.495428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:07.495435 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:07.495486 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:07.499590 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.500470 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.500477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.500480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.500634 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.501229 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.501235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.501238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.501634 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:07.501788 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:07.502581 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 05:03:07.502827 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 05:03:07.504410 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.483475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:44251)
0927 05:03:07.484320 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:07.484360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:07.486222 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:07.486998 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:07.487007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:07.487566 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:07.488142 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:07.488160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.489379 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.489383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:07.490645 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:07.490657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.491493 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.491501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:07.491729 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.492524 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:07.492559 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:07.495106 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:07.499743 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:07.499752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:07.499768 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:07.500339 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:07.500763 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:07.500768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:07.500771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:07.501075 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:07.501772 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:07.501780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:07.502188 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 05:03:07.502474 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 05:03:07.503108 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":347,"threads_started":1}
I20260927 05:03:07.505983 5240 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35719
I20260927 05:03:07.506314 5240 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:07.507284 5240 heartbeater.cc:511] Master 127.20.169.126:35719 requested a full tablet report, sending...
I20260927 05:03:07.508984 4922 ts_manager.cc:194] Registered new tserver with Master: 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847)
I20260927 05:03:07.510113 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:44251
W20260927 05:03:07.819072 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:07.819418 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:07.819586 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:07.826704 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:07.826846 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:07.827031 5242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:03:07.836481 5242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:33927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=39645
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:07.837828 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:07.839109 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:07.852635 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:07.852728 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:07.852686 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:07.854401 5242 server_base.cc:1034] running on GCE node
I20260927 05:03:07.855083 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:07.856148 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:07.857412 5242 hybrid_clock.cc:648] HybridClock initialized: now 1790485387857299 us; error 122 us; skew 500 ppm
I20260927 05:03:07.857793 5242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:07 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485387, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:07.861380 5242 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:03:07.862963 5242 webserver.cc:460] Webserver started at http://127.20.169.67:39645/ using document root <none> and password file <none>
I20260927 05:03:07.863724 5242 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:07.863853 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:07.878890 5242 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.009s sys 0.004s
I20260927 05:03:07.882282 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.883839 5242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:07.884056 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "a41f5d12cfd7496c84daadc1404ae254"
format_stamp: "Formatted at 2026-09-27 05:02:45 on dist-test-slave-39f2"
server_key: "90703abc048903da295ec81eeda2d60e"
server_key_iv: "503c203c8a819dbe5959b9cba64ee489"
server_key_version: "encryptionkey@0"
I20260927 05:03:07.885244 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:07.897857 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:07.902850 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:07.903160 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:07.904223 5242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:07.905813 5242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:07.905938 5242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.906076 5242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:07.906184 5242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:07.943347 5242 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:33927
I20260927 05:03:07.943544 5370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:33927 every 8 connection(s)
I20260927 05:03:07.945166 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:03:07.950970 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5242
Sep 27 05:03:07 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485387, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:07.970011 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.947866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:48837 (local address 127.20.169.126:35719)
0927 05:03:07.948083 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:07.948087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:07.949437 (+ 1350us) server_negotiation.cc:407] Connection header received
0927 05:03:07.951854 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:07.951861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:07.951955 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:07.952073 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:07.954277 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.955436 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.957612 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.958242 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.958443 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.962533 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:07.962537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:07.962540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:07.962585 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:07.966143 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.967074 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.967081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.967085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.967261 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.967893 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.967899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.967902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.968280 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:07.968424 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:07.969434 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 05:03:07.969654 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":88}
I20260927 05:03:07.970042 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.951971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.20.169.126:35719)
0927 05:03:07.952629 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:07.952636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:07.953063 (+ 427us) server_negotiation.cc:407] Connection header received
0927 05:03:07.953318 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:07.953322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:07.953373 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:07.953482 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:07.955471 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.956745 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.958479 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.959297 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.959417 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.962484 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:07.962491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:07.962499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:07.962537 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:07.966542 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.967528 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.967533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.967537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.967695 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:07.968209 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:07.968213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:07.968215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:07.968595 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:07.968903 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:07.969203 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 05:03:07.969507 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":539,"thread_start_us":434,"threads_started":1}
I20260927 05:03:07.971355 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.948389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:48837)
0927 05:03:07.949224 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:07.949313 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:07.951343 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:07.952508 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:07.952526 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:07.953415 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:07.954082 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:07.954101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.955623 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.955632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:07.957463 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:07.957476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.958549 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.958559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:07.958924 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.959754 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:07.959779 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:07.962297 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:07.966289 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:07.966300 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:07.966319 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:07.966913 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:07.967398 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:07.967401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:07.967404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:07.967714 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:07.968422 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:07.968431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:07.968963 (+ 532us) client_negotiation.cc:769] Sending connection context
0927 05:03:07.969305 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 05:03:07.969887 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":381,"threads_started":1}
I20260927 05:03:07.972994 5371 heartbeater.cc:347] Connected to a master server at 127.20.169.126:35719
I20260927 05:03:07.973330 5371 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:07.974141 5371 heartbeater.cc:511] Master 127.20.169.126:35719 requested a full tablet report, sending...
I20260927 05:03:07.975625 4922 ts_manager.cc:194] Registered new tserver with Master: a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927)
I20260927 05:03:07.976472 4922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:48837
I20260927 05:03:07.985090 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:03:07.992058 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.986113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.20.169.126:35719)
0927 05:03:07.986298 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:07.986303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:07.986557 (+ 254us) server_negotiation.cc:407] Connection header received
0927 05:03:07.986731 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:07.986735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:07.986906 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:07.987031 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:03:07.987422 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.988219 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.990108 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.990629 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.990732 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:07.991346 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:03:07.991560 (+ 214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 05:03:07.991850 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50914: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 05:03:07.992210 5374 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50914: token expired
W20260927 05:03:07.992942 4844 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.169.126:35719)
I20260927 05:03:08.006155 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:07.993407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.20.169.126:35719)
0927 05:03:07.993593 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:07.993599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:07.993642 (+ 43us) server_negotiation.cc:407] Connection header received
0927 05:03:07.993833 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:07.993846 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:07.993902 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:07.994001 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:07.995408 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.996111 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.997585 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:07.997996 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:07.998135 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.000685 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:08.000690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:08.000693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:08.000724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:08.003422 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.004079 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.004085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.004088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.004221 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.004616 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.004621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.004626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.005031 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:08.005289 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.005583 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.005843 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:03:08.010035 4922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:03:08.017238 4922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:03:08.045967 5108 heartbeater.cc:503] Master 127.20.169.126:35719 was elected leader, sending a full tablet report...
I20260927 05:03:08.051928 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.029066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.20.169.65:39767)
0927 05:03:08.043458 (+ 14392us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.043466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.043491 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:03:08.043598 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.043609 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.043842 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.044636 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:08.045171 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.046816 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.049208 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.050635 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.050787 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.051029 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.051359 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.051539 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14209,"thread_start_us":14046,"threads_started":1}
I20260927 05:03:08.051936 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.028700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44398)
0927 05:03:08.038688 (+ 9988us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.038753 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:08.038992 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:08.044135 (+ 5143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:08.044142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:08.044172 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:08.044949 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.044960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.046974 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.046979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.049012 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.049028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.050860 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.050866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.051070 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.051130 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 05:03:08.051264 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 05:03:08.051460 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9627,"mutex_wait_us":1,"thread_start_us":12119,"threads_started":1}
I20260927 05:03:08.052140 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.028960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45847 (local address 127.0.0.1:36064)
0927 05:03:08.041073 (+ 12113us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.041097 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:08.041240 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:08.043533 (+ 2293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:08.043541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:08.043570 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:08.044117 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.044132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.046102 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.046106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.048688 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.048704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.051163 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.051168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.051438 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.051472 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:03:08.051560 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 05:03:08.051699 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11743,"mutex_wait_us":84,"thread_start_us":13807,"threads_started":1}
I20260927 05:03:08.052897 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.029216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.20.169.66:45847)
0927 05:03:08.042693 (+ 13477us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.042705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.042745 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:03:08.042856 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.042865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.043168 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.043459 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:08.044435 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.045941 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.049118 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.050635 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.050787 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.050996 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.052356 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.052542 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13272,"thread_start_us":13136,"threads_started":1}
I20260927 05:03:08.052965 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.031336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:33927 (local address 127.0.0.1:54286)
0927 05:03:08.043105 (+ 11769us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.043128 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:08.043247 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:08.043936 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:08.043941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:08.043969 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:08.044917 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.044937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.047909 (+ 2972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.047916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.050131 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.050143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.052070 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.052081 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.052264 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.052296 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:03:08.052373 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 05:03:08.052552 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11614,"mutex_wait_us":4,"spinlock_wait_cycles":3968,"thread_start_us":6829,"threads_started":1}
I20260927 05:03:08.053071 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.032097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.20.169.67:33927)
0927 05:03:08.042153 (+ 10056us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.042166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.043362 (+ 1196us) server_negotiation.cc:407] Connection header received
0927 05:03:08.043498 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.043505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.043726 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.043931 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:08.045326 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.047735 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.050284 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.051533 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.051657 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.051885 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.052526 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.052702 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9816,"thread_start_us":9628,"threads_started":1}
I20260927 05:03:08.056427 5043 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5c5f523ce743ba85f3229b14bd7270 (DEFAULT_TABLE table=test-table [id=85d9d6171e7248a89ee2787264559551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:08.056617 5306 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5c5f523ce743ba85f3229b14bd7270 (DEFAULT_TABLE table=test-table [id=85d9d6171e7248a89ee2787264559551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:08.056613 5175 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5c5f523ce743ba85f3229b14bd7270 (DEFAULT_TABLE table=test-table [id=85d9d6171e7248a89ee2787264559551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:08.057911 5043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5c5f523ce743ba85f3229b14bd7270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:08.058015 5175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5c5f523ce743ba85f3229b14bd7270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:08.058013 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5c5f523ce743ba85f3229b14bd7270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:08.070963 5387 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Bootstrap starting.
I20260927 05:03:08.073467 5388 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Bootstrap starting.
I20260927 05:03:08.074111 5389 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Bootstrap starting.
I20260927 05:03:08.075424 5387 tablet_bootstrap.cc:654] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:08.077234 5387 log.cc:824] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:08.077742 5389 tablet_bootstrap.cc:654] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:08.078676 5388 tablet_bootstrap.cc:654] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:08.079030 5389 log.cc:824] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:08.079782 5388 log.cc:824] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:08.080611 5387 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: No bootstrap required, opened a new log
I20260927 05:03:08.080926 5387 ts_tablet_manager.cc:1397] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 05:03:08.082137 5389 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: No bootstrap required, opened a new log
I20260927 05:03:08.082398 5389 ts_tablet_manager.cc:1397] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:03:08.087240 5388 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: No bootstrap required, opened a new log
I20260927 05:03:08.087550 5388 ts_tablet_manager.cc:1397] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 05:03:08.092576 5387 raft_consensus.cc:348] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.093098 5387 raft_consensus.cc:374] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:08.093227 5387 raft_consensus.cc:729] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90cb00af7d2481bb25845a3ea1ba5b7, State: Initialized, Role: FOLLOWER
I20260927 05:03:08.093181 5389 raft_consensus.cc:348] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.093719 5389 raft_consensus.cc:374] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:08.093842 5389 raft_consensus.cc:729] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41f5d12cfd7496c84daadc1404ae254, State: Initialized, Role: FOLLOWER
I20260927 05:03:08.093937 5387 consensus_queue.cc:260] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.094453 5389 consensus_queue.cc:260] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.096164 5387 ts_tablet_manager.cc:1428] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 05:03:08.095950 5388 raft_consensus.cc:348] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.096452 5388 raft_consensus.cc:374] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:08.096611 5388 raft_consensus.cc:729] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347533a9ed6942bab90e02697f054521, State: Initialized, Role: FOLLOWER
I20260927 05:03:08.097307 5371 heartbeater.cc:503] Master 127.20.169.126:35719 was elected leader, sending a full tablet report...
I20260927 05:03:08.097898 5389 ts_tablet_manager.cc:1428] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260927 05:03:08.098347 5388 consensus_queue.cc:260] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.101475 5240 heartbeater.cc:503] Master 127.20.169.126:35719 was elected leader, sending a full tablet report...
I20260927 05:03:08.102213 5388 ts_tablet_manager.cc:1428] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
W20260927 05:03:08.196878 5372 tablet.cc:2369] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:03:08.224160 5241 tablet.cc:2369] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:03:08.227991 5394 raft_consensus.cc:482] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:08.228292 5394 raft_consensus.cc:504] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.230373 5394 leader_election.cc:288] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65:39767), 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847)
I20260927 05:03:08.235996 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.230540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.20.169.67:45931)
0927 05:03:08.230859 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.230883 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:08.231041 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:08.231436 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:08.231445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:08.231476 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:08.231677 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.231686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.232724 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.232732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.234119 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.234132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.235204 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.235212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.235412 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.235436 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:03:08.235552 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 05:03:08.235689 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 05:03:08.235975 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.230619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:45931 (local address 127.20.169.65:39767)
0927 05:03:08.230965 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.230971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.230992 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:03:08.231210 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.231218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.231286 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.231401 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:08.231864 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.232591 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.234286 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.235048 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.235226 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.235304 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.235582 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.235716 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260927 05:03:08.236790 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.230949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:47471 (local address 127.20.169.66:45847)
0927 05:03:08.231170 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.231182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.231747 (+ 565us) server_negotiation.cc:407] Connection header received
0927 05:03:08.231907 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.231911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.231968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.232066 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:08.232561 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.233323 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.234975 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.235745 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.235922 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.235996 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.236428 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.236572 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:03:08.237195 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c5f523ce743ba85f3229b14bd7270" candidate_uuid: "a41f5d12cfd7496c84daadc1404ae254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" is_pre_election: true
I20260927 05:03:08.237370 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.230873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45847 (local address 127.20.169.67:47471)
0927 05:03:08.231635 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.231652 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:08.231766 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:08.232080 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:08.232084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:08.232106 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:08.232416 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.232425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.233468 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.233474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.234807 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:08.234818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.235960 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.235966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:08.236203 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.236254 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 05:03:08.236997 (+ 743us) client_negotiation.cc:240] Negotiation successful
0927 05:03:08.237137 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":446,"threads_started":1}
I20260927 05:03:08.237735 5063 raft_consensus.cc:2454] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41f5d12cfd7496c84daadc1404ae254 in term 0.
I20260927 05:03:08.238433 5259 leader_election.cc:302] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41f5d12cfd7496c84daadc1404ae254, d90cb00af7d2481bb25845a3ea1ba5b7; no voters:
I20260927 05:03:08.238375 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c5f523ce743ba85f3229b14bd7270" candidate_uuid: "a41f5d12cfd7496c84daadc1404ae254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347533a9ed6942bab90e02697f054521" is_pre_election: true
I20260927 05:03:08.238830 5195 raft_consensus.cc:2454] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41f5d12cfd7496c84daadc1404ae254 in term 0.
I20260927 05:03:08.238857 5394 raft_consensus.cc:2781] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:03:08.239017 5394 raft_consensus.cc:482] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:03:08.239135 5394 raft_consensus.cc:3037] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:08.241482 5394 raft_consensus.cc:504] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.242175 5394 leader_election.cc:288] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [CANDIDATE]: Term 1 election: Requested vote from peers d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65:39767), 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847)
I20260927 05:03:08.242676 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c5f523ce743ba85f3229b14bd7270" candidate_uuid: "a41f5d12cfd7496c84daadc1404ae254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7"
I20260927 05:03:08.242733 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c5f523ce743ba85f3229b14bd7270" candidate_uuid: "a41f5d12cfd7496c84daadc1404ae254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347533a9ed6942bab90e02697f054521"
I20260927 05:03:08.242884 5063 raft_consensus.cc:3037] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:08.242985 5195 raft_consensus.cc:3037] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:08.245536 5063 raft_consensus.cc:2454] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41f5d12cfd7496c84daadc1404ae254 in term 1.
I20260927 05:03:08.245810 5195 raft_consensus.cc:2454] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41f5d12cfd7496c84daadc1404ae254 in term 1.
I20260927 05:03:08.245971 5259 leader_election.cc:302] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41f5d12cfd7496c84daadc1404ae254, d90cb00af7d2481bb25845a3ea1ba5b7; no voters:
I20260927 05:03:08.246309 5394 raft_consensus.cc:2781] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:08.246768 5394 raft_consensus.cc:686] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 1 LEADER]: Becoming Leader. State: Replica: a41f5d12cfd7496c84daadc1404ae254, State: Running, Role: LEADER
I20260927 05:03:08.247326 5394 consensus_queue.cc:237] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:08.252187 4921 catalog_manager.cc:5495] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "a41f5d12cfd7496c84daadc1404ae254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } health_report { overall_health: HEALTHY } } }
W20260927 05:03:08.257612 5109 tablet.cc:2369] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:03:08.290638 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.284300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.20.169.67:33927)
0927 05:03:08.284496 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.284501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.284523 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:03:08.284692 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.284697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.284838 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.285009 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:03:08.285457 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.286461 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.287863 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.288496 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.288627 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.289303 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:03:08.290063 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:03:08.290221 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.290278 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.290366 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:03:08.301340 5063 raft_consensus.cc:1261] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Refusing update from remote peer a41f5d12cfd7496c84daadc1404ae254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:08.301470 5195 raft_consensus.cc:1261] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Refusing update from remote peer a41f5d12cfd7496c84daadc1404ae254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:08.302589 5397 consensus_queue.cc:1035] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:08.302986 5394 consensus_queue.cc:1035] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:08.318676 5402 mvcc.cc:205] Tried to move back new op lower bound from 7333828150471147520 to 7333828150265729024. Current Snapshot: MvccSnapshot[applied={T|T < 7333828150471147520}]
I20260927 05:03:08.318944 5400 mvcc.cc:205] Tried to move back new op lower bound from 7333828150471147520 to 7333828150265729024. Current Snapshot: MvccSnapshot[applied={T|T < 7333828150471147520}]
I20260927 05:03:08.330423 4921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:08.330719 4921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:08.334403 4921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Sending DeleteTablet for 3 replicas of tablet 2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:08.335515 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5c5f523ce743ba85f3229b14bd7270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:08 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:54286
I20260927 05:03:08.335511 5175 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5c5f523ce743ba85f3229b14bd7270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:08 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:36064
I20260927 05:03:08.335554 5043 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a5c5f523ce743ba85f3229b14bd7270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:08 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:44398
I20260927 05:03:08.337515 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4959
I20260927 05:03:08.370684 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.338531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.20.169.65:39767)
0927 05:03:08.339175 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.339184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.339207 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:03:08.339279 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.339298 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.339398 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.339494 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:08.342082 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.343699 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.346361 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.347315 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.347496 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.357832 (+ 10336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:08.357859 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:08.357887 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:08.357948 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:08.363410 (+ 5462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.364713 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.364726 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.364733 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.364983 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.366622 (+ 1639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.366630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.366635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.367247 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:08.367658 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.369721 (+ 2063us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.370066 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260927 05:03:08.373231 5103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:08.391615 5410 tablet_replica.cc:330] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: stopping tablet replica
I20260927 05:03:08.394028 5410 raft_consensus.cc:2229] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:03:08.394963 5410 raft_consensus.cc:2258] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:08.396323 5411 tablet_replica.cc:330] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: stopping tablet replica
I20260927 05:03:08.396745 5411 raft_consensus.cc:2229] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:08.397313 5411 raft_consensus.cc:2258] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:08.397642 5410 ts_tablet_manager.cc:1905] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:08.399364 5411 ts_tablet_manager.cc:1905] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:08.405303 5410 ts_tablet_manager.cc:1918] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:08.405552 5410 log.cc:1196] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:08.406268 5410 ts_tablet_manager.cc:1939] T 2a5c5f523ce743ba85f3229b14bd7270 P a41f5d12cfd7496c84daadc1404ae254: Deleting consensus metadata
I20260927 05:03:08.406415 5411 ts_tablet_manager.cc:1918] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:08.406617 5411 log.cc:1196] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:08.407213 5411 ts_tablet_manager.cc:1939] T 2a5c5f523ce743ba85f3229b14bd7270 P 347533a9ed6942bab90e02697f054521: Deleting consensus metadata
I20260927 05:03:08.408524 4907 catalog_manager.cc:4841] TS a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927): tablet 2a5c5f523ce743ba85f3229b14bd7270 (table test-table [id=85d9d6171e7248a89ee2787264559551]) successfully deleted
I20260927 05:03:08.409271 4909 catalog_manager.cc:4841] TS 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847): tablet 2a5c5f523ce743ba85f3229b14bd7270 (table test-table [id=85d9d6171e7248a89ee2787264559551]) successfully deleted
I20260927 05:03:08.550011 5412 tablet_replica.cc:330] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: stopping tablet replica
I20260927 05:03:08.550446 5412 raft_consensus.cc:2229] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:08.550946 5412 raft_consensus.cc:2258] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:08.551681 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4959
I20260927 05:03:08.571215 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 5112
W20260927 05:03:08.571208 4906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:39767 (error 108)
I20260927 05:03:08.574023 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.572766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44420)
0927 05:03:08.573654 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.573802 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":668,"threads_started":1}
W20260927 05:03:08.574331 4906 catalog_manager.cc:4584] TS d90cb00af7d2481bb25845a3ea1ba5b7 (127.20.169.65:39767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a5c5f523ce743ba85f3229b14bd7270: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
I20260927 05:03:08.590816 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.572006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.20.169.66:45847)
0927 05:03:08.572196 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.572203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.572886 (+ 683us) server_negotiation.cc:407] Connection header received
0927 05:03:08.572952 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.572956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.573019 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.573133 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:08.575125 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.575864 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.577401 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.577805 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.577946 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.584532 (+ 6586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:08.584546 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:08.584561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:08.584603 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:08.587642 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.588359 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.588367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.588369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.588495 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.588959 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.588963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.588965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.589454 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:08.589772 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.590090 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.590445 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 05:03:08.591500 5235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:08.628001 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.627448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44436)
0927 05:03:08.627696 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.627817 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 05:03:08.629256 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.628827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44440)
0927 05:03:08.629003 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.629096 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 05:03:08.685269 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 5112
I20260927 05:03:08.690441 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.689765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44446)
0927 05:03:08.690045 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.690163 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:03:08.691809 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.691303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44450)
0927 05:03:08.691514 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.691619 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:03:08.705785 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 5242
I20260927 05:03:08.723516 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.706750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.20.169.67:33927)
0927 05:03:08.706964 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:08.706979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:08.707011 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:03:08.707164 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:08.707171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:08.707274 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:08.707394 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:08.708766 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.710055 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.711482 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:08.712191 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:08.712339 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:08.715051 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:08.715071 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:08.715093 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:08.715143 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:08.719813 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.720659 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.720670 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.720674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.720956 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:08.721593 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:08.721599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:08.721603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:08.722106 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:08.722466 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:08.722770 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 05:03:08.723013 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:03:08.724363 5366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:08.772665 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.772124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44466)
0927 05:03:08.772372 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.772486 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 05:03:08.773988 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.773514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44476)
0927 05:03:08.773724 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.773817 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 05:03:08.852756 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 5242
I20260927 05:03:08.873569 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 4889
I20260927 05:03:08.874799 4952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:08.987352 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:08.986905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:39767 (local address 127.0.0.1:44486)
0927 05:03:08.987098 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:08.987198 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:03:08.987586 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 4889
I20260927 05:03:29.011454 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35719
--webserver_interface=127.20.169.126
--webserver_port=43097
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:29.349838 5419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:29.350222 5419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:29.350390 5419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:29.357705 5419 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:03:29.357839 5419 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:03:29.357928 5419 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:03:29.358026 5419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:03:29.358126 5419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:29.358232 5419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:29.358316 5419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:03:29.358395 5419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:03:29.368435 5419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=43097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:29.369850 5419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:29.371258 5419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:29.383570 5424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:29.383606 5427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:29.383680 5425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:29.384377 5419 server_base.cc:1034] running on GCE node
I20260927 05:03:29.385205 5419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:29.386670 5419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:29.387923 5419 hybrid_clock.cc:648] HybridClock initialized: now 1790485409387853 us; error 48 us; skew 500 ppm
I20260927 05:03:29.388257 5419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:29 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485409, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:29.391415 5419 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:03:29.392974 5419 webserver.cc:460] Webserver started at http://127.20.169.126:43097/ using document root <none> and password file <none>
I20260927 05:03:29.393625 5419 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:29.393713 5419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:29.399086 5419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 05:03:29.402011 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:29.403311 5419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:03:29.403486 5419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "4554ab0503bd4ac781a5a08cc424688d"
format_stamp: "Formatted at 2026-09-27 05:02:43 on dist-test-slave-39f2"
server_key: "919b481ccccc877f63d0d3df40692d55"
server_key_iv: "1c65fadb430258f2ba98e90188e7b4f7"
server_key_version: "encryptionkey@0"
I20260927 05:03:29.404446 5419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:29.419032 5419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:29.423097 5419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:29.423339 5419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:29.442814 5419 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35719
I20260927 05:03:29.443004 5485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35719 every 8 connection(s)
I20260927 05:03:29.444378 5419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:03:29.447264 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5419
I20260927 05:03:29.448127 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:39767
--local_ip_for_outbound_sockets=127.20.169.65
--tserver_master_addrs=127.20.169.126:35719
--webserver_port=44541
--webserver_interface=127.20.169.65
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:03:29.452011 5486 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:03:29.455549 5486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap starting.
I20260927 05:03:29.488962 5486 log.cc:824] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:29.508497 5486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:03:29.509147 5486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Bootstrap complete.
I20260927 05:03:29.518360 5486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:29.519277 5486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4554ab0503bd4ac781a5a08cc424688d, State: Initialized, Role: FOLLOWER
I20260927 05:03:29.519915 5486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:29.520172 5486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:03:29.520314 5486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:03:29.520499 5486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:03:29.524273 5486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:29.524760 5486 leader_election.cc:302] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4554ab0503bd4ac781a5a08cc424688d; no voters:
I20260927 05:03:29.525525 5486 leader_election.cc:288] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 05:03:29.525666 5490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:03:29.527010 5490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [term 3 LEADER]: Becoming Leader. State: Replica: 4554ab0503bd4ac781a5a08cc424688d, State: Running, Role: LEADER
I20260927 05:03:29.527638 5490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } }
I20260927 05:03:29.527829 5486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:03:29.530277 5492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4554ab0503bd4ac781a5a08cc424688d. Latest consensus state: current_term: 3 leader_uuid: "4554ab0503bd4ac781a5a08cc424688d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } } }
I20260927 05:03:29.530687 5492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:29.531373 5491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4554ab0503bd4ac781a5a08cc424688d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4554ab0503bd4ac781a5a08cc424688d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35719 } } }
I20260927 05:03:29.531680 5491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:29.536219 5497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:03:29.546562 5497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:03:29.548539 5497 catalog_manager.cc:1259] Loaded cluster ID: a4ed0ffd958e49f385a82832602e6c05
I20260927 05:03:29.548728 5497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:03:29.554106 5497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:03:29.556280 5497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Loaded TSK: 0
I20260927 05:03:29.556558 5497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4554ab0503bd4ac781a5a08cc424688d: Loaded TSK: 1
F20260927 05:03:29.558368 5497 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7f1bfaf9ad google::LogMessage::Fail() at ??:0
@ 0x7f7f1bfb288a google::LogMessage::SendToLog() at ??:0
@ 0x7f7f1bfaf4de google::LogMessage::Flush() at ??:0
@ 0x7f7f1bfb0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7f1fd137d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7f1fd0273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7f319de1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7f319dd783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7f31a692a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7f31a690c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7f31ab35e9 std::function<>::operator()() at ??:0
@ 0x7f7f319e1cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7f319dfc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260927 05:03:29.617511 5509 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f7f31a67610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7f31a67439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7f1ebfb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7f1ebfe91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7f1ebfe799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7f1ebd9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7f21eb36db start_thread at ??:0
@ 0x7f7f1ae4961f clone at ??:0
W20260927 05:03:29.888264 5488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:29.888541 5488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:29.888643 5488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:29.896138 5488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:29.896261 5488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:29.896400 5488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:03:29.906423 5488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:39767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=44541
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:29.907729 5488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:29.909143 5488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:29.923571 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:29.923631 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:29.923681 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:29.924729 5488 server_base.cc:1034] running on GCE node
I20260927 05:03:29.925444 5488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:29.926390 5488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:29.927618 5488 hybrid_clock.cc:648] HybridClock initialized: now 1790485409927533 us; error 57 us; skew 500 ppm
I20260927 05:03:29.927966 5488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:29 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485409, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:29.931207 5488 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:03:29.932744 5488 webserver.cc:460] Webserver started at http://127.20.169.65:44541/ using document root <none> and password file <none>
I20260927 05:03:29.933475 5488 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:29.933625 5488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:29.938953 5488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 05:03:29.941828 5543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:29.943210 5488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:03:29.943423 5488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "d90cb00af7d2481bb25845a3ea1ba5b7"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "2557120bfb4fe81635c814600f10cb2e"
server_key_iv: "c0a7ef69bb78349a9d85bc2fda70c784"
server_key_version: "encryptionkey@0"
I20260927 05:03:29.944360 5488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:29.960876 5488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:29.965353 5488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:29.965641 5488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:29.966645 5488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:29.968443 5550 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 05:03:29.974301 5488 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 05:03:29.974421 5488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 05:03:29.974514 5488 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 05:03:29.977670 5488 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 05:03:29.977756 5488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:29.978164 5550 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Bootstrap starting.
I20260927 05:03:30.005101 5550 log.cc:824] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:30.019462 5550 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:03:30.020037 5550 tablet_bootstrap.cc:492] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Bootstrap complete.
I20260927 05:03:30.021618 5550 ts_tablet_manager.cc:1397] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Time spent bootstrapping tablet: real 0.044s user 0.027s sys 0.015s
I20260927 05:03:30.026467 5488 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:39767
I20260927 05:03:30.026588 5657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:39767 every 8 connection(s)
I20260927 05:03:30.028030 5488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:03:30.030134 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5488
I20260927 05:03:30.030812 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:45847
--local_ip_for_outbound_sockets=127.20.169.66
--tserver_master_addrs=127.20.169.126:35719
--webserver_port=40757
--webserver_interface=127.20.169.66
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:03:30.032967 5550 raft_consensus.cc:348] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:30.035055 5550 raft_consensus.cc:729] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90cb00af7d2481bb25845a3ea1ba5b7, State: Initialized, Role: FOLLOWER
I20260927 05:03:30.035863 5550 consensus_queue.cc:260] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:30.037917 5550 ts_tablet_manager.cc:1428] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7: Time spent starting tablet: real 0.016s user 0.011s sys 0.001s
I20260927 05:03:30.038199 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.035259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:58937)
0927 05:03:30.036360 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.036765 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":403,"threads_started":1}
W20260927 05:03:30.038944 5658 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:35719 (0 consecutive failures): Network error: Failed to ping master at 127.20.169.126:35719: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
I20260927 05:03:30.040043 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.039543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:46113)
0927 05:03:30.039749 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.039859 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:03:30.041098 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.040613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:53143)
0927 05:03:30.040787 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.040917 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:03:30.041491 5658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:03:30.360194 5662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:30.360458 5662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:30.360616 5662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:30.367708 5662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:30.367806 5662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:30.368017 5662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:03:30.377418 5662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:45847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=40757
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:30.378698 5662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:30.379976 5662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:30.393282 5668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:30.393683 5667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:30.393433 5670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:30.395082 5662 server_base.cc:1034] running on GCE node
I20260927 05:03:30.395848 5662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:30.397038 5662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:30.398326 5662 hybrid_clock.cc:648] HybridClock initialized: now 1790485410398249 us; error 51 us; skew 500 ppm
I20260927 05:03:30.398710 5662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:30 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485410, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:30.402268 5662 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:03:30.403729 5662 webserver.cc:460] Webserver started at http://127.20.169.66:40757/ using document root <none> and password file <none>
I20260927 05:03:30.404420 5662 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:30.404531 5662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:30.410406 5662 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 05:03:30.413473 5677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:30.414898 5662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:30.415076 5662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "347533a9ed6942bab90e02697f054521"
format_stamp: "Formatted at 2026-09-27 05:02:44 on dist-test-slave-39f2"
server_key: "ca6cd05c21c6555e554af35768f1fa04"
server_key_iv: "b7ab26dcb7aadb41a738bc7fa94306ac"
server_key_version: "encryptionkey@0"
I20260927 05:03:30.416136 5662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:30.446955 5662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:30.451894 5662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:30.452179 5662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:30.453269 5662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:30.454648 5662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:30.454734 5662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:30.454871 5662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:30.454952 5662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:30.491421 5662 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:45847
I20260927 05:03:30.491518 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:45847 every 8 connection(s)
I20260927 05:03:30.492949 5662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:03:30.497884 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.495965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:51835)
0927 05:03:30.496665 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.496997 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":262,"threads_started":1}
W20260927 05:03:30.498502 5790 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:35719 (0 consecutive failures): Network error: Failed to ping master at 127.20.169.126:35719: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
I20260927 05:03:30.498943 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5662
I20260927 05:03:30.499450 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.498973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:51789)
0927 05:03:30.499189 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.499266 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 05:03:30.499779 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:33927
--local_ip_for_outbound_sockets=127.20.169.67
--tserver_master_addrs=127.20.169.126:35719
--webserver_port=39645
--webserver_interface=127.20.169.67
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:03:30.500519 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.500062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:34891)
0927 05:03:30.500222 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.500361 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 05:03:30.500778 5790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:03:30.839572 5793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:30.839893 5793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:30.840057 5793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:30.847546 5793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:30.847705 5793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:30.847873 5793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:03:30.857769 5793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:33927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=39645
--tserver_master_addrs=127.20.169.126:35719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:30.859053 5793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:30.860380 5793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:30.876902 5801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:30.877014 5799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:30.877579 5793 server_base.cc:1034] running on GCE node
W20260927 05:03:30.877691 5798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:30.878547 5793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:30.879462 5793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:30.880695 5793 hybrid_clock.cc:648] HybridClock initialized: now 1790485410880631 us; error 55 us; skew 500 ppm
I20260927 05:03:30.881036 5793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:30 dist-test-slave-39f2 krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485410, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:30.884243 5793 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:03:30.885847 5793 webserver.cc:460] Webserver started at http://127.20.169.67:39645/ using document root <none> and password file <none>
I20260927 05:03:30.886525 5793 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:30.886791 5793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:30.892572 5793 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:03:30.895555 5808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:30.897217 5793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:30.897413 5793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "a41f5d12cfd7496c84daadc1404ae254"
format_stamp: "Formatted at 2026-09-27 05:02:45 on dist-test-slave-39f2"
server_key: "90703abc048903da295ec81eeda2d60e"
server_key_iv: "503c203c8a819dbe5959b9cba64ee489"
server_key_version: "encryptionkey@0"
I20260927 05:03:30.898427 5793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:30.924635 5793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:30.929198 5793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:30.929486 5793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:30.930631 5793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:30.932006 5793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:30.932129 5793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:30.932263 5793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:30.932350 5793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:30.965977 5793 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:33927
I20260927 05:03:30.966064 5920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:33927 every 8 connection(s)
I20260927 05:03:30.967499 5793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:03:30.972303 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.970553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:35011)
0927 05:03:30.971255 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.971504 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
W20260927 05:03:30.972996 5921 heartbeater.cc:644] Failed to heartbeat to 127.20.169.126:35719 (0 consecutive failures): Network error: Failed to ping master at 127.20.169.126:35719: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
I20260927 05:03:30.973778 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.973495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:42645)
0927 05:03:30.973622 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.973684 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 05:03:30.974653 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:30.974367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:58123)
0927 05:03:30.974486 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:30.974553 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 05:03:30.974655 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5793
W20260927 05:03:30.974853 5921 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:03:31.042861 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.042070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:38965)
0927 05:03:31.042554 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:31.042649 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":244,"threads_started":1}
I20260927 05:03:31.502475 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.501350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:52971)
0927 05:03:31.501978 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:31.502120 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":304,"threads_started":1}
I20260927 05:03:31.665993 5927 raft_consensus.cc:482] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:31.666383 5927 raft_consensus.cc:504] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:31.668043 5927 leader_election.cc:288] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847), a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927)
Sep 27 05:03:31 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485409, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
Sep 27 05:03:31 dist-test-slave-39f2 krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485409, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:03:31.708477 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.668624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:59625 (local address 127.20.169.66:45847)
0927 05:03:31.681069 (+ 12445us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:31.681097 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:31.682580 (+ 1483us) server_negotiation.cc:407] Connection header received
0927 05:03:31.685096 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:31.685104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:31.685590 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:31.685967 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:31.689014 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.690978 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.694025 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.695126 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.695378 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:31.700654 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:31.700667 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:31.700691 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:31.700744 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:31.704302 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:31.705381 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:31.705388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:31.705394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:31.705564 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:31.706074 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:31.706085 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:31.706088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:31.706654 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:31.706866 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:31.707673 (+ 807us) server_negotiation.cc:299] Negotiation successful
0927 05:03:31.708055 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12125,"thread_start_us":12488,"threads_started":1}
I20260927 05:03:31.708488 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.668742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:43143 (local address 127.20.169.67:33927)
0927 05:03:31.682693 (+ 13951us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:31.682703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:31.682737 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:03:31.685036 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:31.685050 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:31.685590 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:31.685967 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:31.689013 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.690978 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.694025 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.695126 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.695329 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:31.701326 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:31.701359 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:31.701378 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:31.701439 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:31.704433 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:31.705342 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:31.705351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:31.705361 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:31.705536 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:31.706049 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:31.706054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:31.706058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:31.706647 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:31.706838 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:31.707674 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 05:03:31.708058 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13781,"thread_start_us":13653,"threads_started":1}
I20260927 05:03:31.708505 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.668244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:45847 (local address 127.20.169.65:59625)
0927 05:03:31.681844 (+ 13600us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:31.681899 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:31.682582 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:31.686150 (+ 3568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:31.686162 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:31.687397 (+ 1235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:31.688740 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:31.688769 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.691212 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.691218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:31.693859 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:31.693876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.695429 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.695435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:31.695882 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:31.696925 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:31.696966 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:31.700265 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:31.704459 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:31.704471 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:31.704489 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:31.705119 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:31.705703 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:31.705707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:31.705710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:31.705934 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:31.706833 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:31.706838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:31.707108 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 05:03:31.707426 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 05:03:31.707998 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13464,"mutex_wait_us":583,"spinlock_wait_cycles":96512,"thread_start_us":13319,"threads_started":1}
I20260927 05:03:31.708566 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.668528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:33927 (local address 127.20.169.65:43143)
0927 05:03:31.682264 (+ 13736us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:31.682281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:31.682631 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:31.686150 (+ 3519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:31.686162 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:31.687386 (+ 1224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:31.688740 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:31.688769 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.691214 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.691218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:31.693859 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:31.693876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:31.695408 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:31.695417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:31.695884 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:31.696925 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:31.696967 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:31.701040 (+ 4073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:31.704578 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:31.704582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:31.704585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:31.705094 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:31.705657 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:31.705665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:31.705668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:31.705928 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:31.706798 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:31.706806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:31.707109 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 05:03:31.707427 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 05:03:31.708149 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13603,"mutex_wait_us":685,"spinlock_wait_cycles":70272,"thread_start_us":13413,"threads_started":1}
W20260927 05:03:31.711761 5545 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
W20260927 05:03:31.712098 5547 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:31.712287 5547 leader_election.cc:302] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d90cb00af7d2481bb25845a3ea1ba5b7; no voters: 347533a9ed6942bab90e02697f054521, a41f5d12cfd7496c84daadc1404ae254
I20260927 05:03:31.712906 5927 raft_consensus.cc:2726] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:03:31.976387 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:31.975361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:57527)
0927 05:03:31.975928 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:31.976163 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":297,"threads_started":1}
I20260927 05:03:32.044387 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:32.043870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:52963)
0927 05:03:32.044041 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:32.044211 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 05:03:32.504101 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:32.503463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:40575)
0927 05:03:32.503873 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:32.503951 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":221,"threads_started":1}
I20260927 05:03:32.977968 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:32.977271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:57859)
0927 05:03:32.977705 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:32.977813 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260927 05:03:33.046248 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:33.045401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:53339)
0927 05:03:33.045969 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:33.046061 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":233,"threads_started":1}
I20260927 05:03:33.506295 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:33.505085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:44989)
0927 05:03:33.505979 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:33.506097 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":356,"threads_started":1}
I20260927 05:03:33.823376 5937 raft_consensus.cc:482] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:33.823704 5937 raft_consensus.cc:504] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:33.824481 5937 leader_election.cc:288] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847), a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927)
W20260927 05:03:33.825575 5545 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
W20260927 05:03:33.825816 5547 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:33.825961 5547 leader_election.cc:302] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d90cb00af7d2481bb25845a3ea1ba5b7; no voters: 347533a9ed6942bab90e02697f054521, a41f5d12cfd7496c84daadc1404ae254
I20260927 05:03:33.826234 5937 raft_consensus.cc:2726] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:03:33.979769 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:33.979001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:41085)
0927 05:03:33.979517 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:33.979606 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":255,"threads_started":1}
I20260927 05:03:34.048525 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:34.047385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:34301)
0927 05:03:34.048066 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:34.048225 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
I20260927 05:03:34.507913 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:34.507129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:54049)
0927 05:03:34.507625 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:34.507725 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260927 05:03:34.981554 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:34.980771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:42833)
0927 05:03:34.981297 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:34.981376 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260927 05:03:35.050483 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:35.049694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:42473)
0927 05:03:35.050198 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:35.050288 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":255,"threads_started":1}
I20260927 05:03:35.509732 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:35.508921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:45467)
0927 05:03:35.509464 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:35.509557 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":286,"threads_started":1}
I20260927 05:03:35.983373 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:35.982523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:45537)
0927 05:03:35.983089 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:35.983174 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":344,"threads_started":1}
I20260927 05:03:36.052304 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:36.051499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:44057)
0927 05:03:36.052020 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:36.052114 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260927 05:03:36.511745 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:36.510798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:53111)
0927 05:03:36.511387 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:36.511508 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":318,"threads_started":1}
I20260927 05:03:36.985105 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:36.984301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:58137)
0927 05:03:36.984780 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:36.984928 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260927 05:03:37.053953 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:37.053207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:44175)
0927 05:03:37.053649 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:37.053736 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260927 05:03:37.065881 5949 raft_consensus.cc:482] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:37.066120 5949 raft_consensus.cc:504] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:37.066897 5949 leader_election.cc:288] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847), a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927)
W20260927 05:03:37.067890 5547 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
W20260927 05:03:37.068101 5545 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:37.068271 5545 leader_election.cc:302] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d90cb00af7d2481bb25845a3ea1ba5b7; no voters: 347533a9ed6942bab90e02697f054521, a41f5d12cfd7496c84daadc1404ae254
I20260927 05:03:37.068542 5949 raft_consensus.cc:2726] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:03:37.513643 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:37.512709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:42239)
0927 05:03:37.513343 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:37.513442 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":315,"threads_started":1}
I20260927 05:03:37.986912 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:37.986075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:58461)
0927 05:03:37.986604 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:37.986692 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":241,"threads_started":1}
I20260927 05:03:38.055907 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:38.055041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:44757)
0927 05:03:38.055586 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:38.055680 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":303,"threads_started":1}
I20260927 05:03:38.515468 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:38.514671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:39145)
0927 05:03:38.515166 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:38.515268 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":270,"threads_started":1}
I20260927 05:03:38.988613 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:38.987898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:58397)
0927 05:03:38.988353 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:38.988432 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":249,"threads_started":1}
I20260927 05:03:39.057626 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:39.056880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:46351)
0927 05:03:39.057367 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:39.057451 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
I20260927 05:03:39.517278 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:39.516464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:58173)
0927 05:03:39.517031 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:39.517122 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
I20260927 05:03:39.990386 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:39.989521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:59507)
0927 05:03:39.990090 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:39.990207 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":275,"threads_started":1}
I20260927 05:03:40.059408 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:40.058659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:46041)
0927 05:03:40.059154 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:40.059242 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260927 05:03:40.519006 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:40.518279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:51383)
0927 05:03:40.518751 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:40.518833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":286,"threads_started":1}
I20260927 05:03:40.992060 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:40.991344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:52381)
0927 05:03:40.991774 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:40.991851 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260927 05:03:41.061169 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:41.060363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:49683)
0927 05:03:41.060878 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:41.060978 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260927 05:03:41.521286 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:41.520111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:37285)
0927 05:03:41.520866 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:41.521009 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":382,"threads_started":1}
I20260927 05:03:41.958838 5963 raft_consensus.cc:482] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:41.959071 5963 raft_consensus.cc:504] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90cb00af7d2481bb25845a3ea1ba5b7" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 39767 } } peers { permanent_uuid: "347533a9ed6942bab90e02697f054521" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 45847 } } peers { permanent_uuid: "a41f5d12cfd7496c84daadc1404ae254" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 33927 } }
I20260927 05:03:41.959766 5963 leader_election.cc:288] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847), a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927)
W20260927 05:03:41.960760 5545 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a41f5d12cfd7496c84daadc1404ae254 (127.20.169.67:33927): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
W20260927 05:03:41.960978 5547 leader_election.cc:341] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 347533a9ed6942bab90e02697f054521 (127.20.169.66:45847): Not found: Tablet not found: 2a5c5f523ce743ba85f3229b14bd7270
I20260927 05:03:41.961161 5547 leader_election.cc:302] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d90cb00af7d2481bb25845a3ea1ba5b7; no voters: 347533a9ed6942bab90e02697f054521, a41f5d12cfd7496c84daadc1404ae254
I20260927 05:03:41.961438 5963 raft_consensus.cc:2726] T 2a5c5f523ce743ba85f3229b14bd7270 P d90cb00af7d2481bb25845a3ea1ba5b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:03:41.993681 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:41.993009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:49335)
0927 05:03:41.993443 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:41.993530 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260927 05:03:42.063004 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:42.062119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:47995)
0927 05:03:42.062639 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:42.062752 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":270,"threads_started":1}
I20260927 05:03:42.523185 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:42.522308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:53375)
0927 05:03:42.522888 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:42.523002 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":277,"threads_started":1}
I20260927 05:03:42.995280 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:42.994557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:58129)
0927 05:03:42.995030 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:42.995114 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":247,"threads_started":1}
I20260927 05:03:43.064877 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:43.064012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:35811)
0927 05:03:43.064508 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:43.064599 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":279,"threads_started":1}
I20260927 05:03:43.524920 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:43.524081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:36299)
0927 05:03:43.524577 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:43.524688 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":279,"threads_started":1}
I20260927 05:03:43.997035 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:43.996270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:52945)
0927 05:03:43.996719 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:43.996867 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260927 05:03:44.066620 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:44.065908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:53521)
0927 05:03:44.066351 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:44.066444 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":222,"threads_started":1}
I20260927 05:03:44.526772 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:44.525989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:52077)
0927 05:03:44.526480 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:44.526567 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":271,"threads_started":1}
I20260927 05:03:44.998869 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:44.997993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:52679)
0927 05:03:44.998533 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:44.998653 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260927 05:03:45.068660 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:45.067821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.65:47791)
0927 05:03:45.068325 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:45.068422 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260927 05:03:45.528858 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:45.527965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.66:44559)
0927 05:03:45.528509 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:45.528620 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":262,"threads_started":1}
I20260927 05:03:45.978178 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 5488
I20260927 05:03:45.998481 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:45.979247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.20.169.65:39767)
0927 05:03:45.979966 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:45.979994 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:45.980031 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:03:45.980131 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:45.980143 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:45.980524 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:45.980732 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:45.982520 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:45.983572 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:45.984912 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:45.985520 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:45.985644 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:45.992352 (+ 6708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:45.992368 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:45.992384 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:45.992434 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:45.995696 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:45.996365 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:45.996373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:45.996375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:45.996521 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:45.996920 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:45.996924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:45.996927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:45.997309 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:45.997558 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:45.997849 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 05:03:45.998117 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":376,"threads_started":1}
I20260927 05:03:45.999816 5653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:46.001209 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:45.999949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:35719 (local address 127.20.169.67:37781)
0927 05:03:46.000729 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:46.000933 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.126:35719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":358,"threads_started":1}
I20260927 05:03:46.093623 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 5488
I20260927 05:03:46.110570 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 5662
I20260927 05:03:46.124214 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:46.111444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.20.169.66:45847)
0927 05:03:46.112011 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:46.112015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:46.112033 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:03:46.112089 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:46.112093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:46.112152 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:46.112246 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:46.113714 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:46.114589 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:46.115848 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:46.116310 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:46.116414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:46.119098 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:46.119104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:46.119107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:46.119140 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:46.122019 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:46.122638 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:46.122641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:46.122644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:46.122761 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:46.123069 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:46.123072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:46.123074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:46.123333 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:46.123524 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:46.123758 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 05:03:46.123921 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":337,"threads_started":1}
I20260927 05:03:46.124914 5785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:46.208762 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 5662
I20260927 05:03:46.227818 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 5793
I20260927 05:03:46.241392 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:46.228675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.20.169.67:33927)
0927 05:03:46.229299 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:46.229304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:46.229321 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:03:46.229379 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:46.229383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:46.229452 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:46.229543 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:46.230865 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:46.231634 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:46.232802 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:46.233311 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:46.233411 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:46.235911 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:46.235916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:46.235919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:46.235950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:46.238733 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:46.239373 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:46.239377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:46.239379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:46.239518 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:46.239868 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:46.239871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:46.239873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:46.240126 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:46.240389 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:46.240619 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 05:03:46.240898 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":345,"threads_started":1}
I20260927 05:03:46.242072 5916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:46.325284 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 5793
2026-09-27T05:03:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64666 ms)
[----------] 30 tests from SecurityITest (300167 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:03:46 dist-test-slave-39f2 krb5kdc[5986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:03:46 dist-test-slave-39f2 krb5kdc[5986](info): set up 2 sockets
Sep 27 05:03:46 dist-test-slave-39f2 krb5kdc[5986](info): commencing operation
krb5kdc: starting...
W20260927 05:03:47.448385 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:03:47 dist-test-slave-39f2 krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:03:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:03:47.709352 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:34193
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:34193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:48.042819 5999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:48.043100 5999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:48.043295 5999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:48.050503 5999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:03:48.050638 5999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:48.050724 5999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:48.050818 5999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:03:48.050871 5999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:03:48.060889 5999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:34193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:34193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:48.062225 5999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:48.063596 5999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:48.075614 6007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:48.075644 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:48.075883 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:48.076448 5999 server_base.cc:1034] running on GCE node
I20260927 05:03:48.077280 5999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:48.078665 5999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:48.079885 5999 hybrid_clock.cc:648] HybridClock initialized: now 1790485428079839 us; error 27 us; skew 500 ppm
I20260927 05:03:48.080243 5999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:48 dist-test-slave-39f2 krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485428, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:48.083549 5999 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:03:48.085136 5999 webserver.cc:460] Webserver started at http://127.20.169.126:37243/ using document root <none> and password file <none>
I20260927 05:03:48.085817 5999 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:48.085947 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:48.086283 5999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:48.089452 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "108d4327bf4a490e8dc78f356ea5ed40"
format_stamp: "Formatted at 2026-09-27 05:03:48 on dist-test-slave-39f2"
server_key: "f45fea1d69e11a2fab80f4d12cd37494"
server_key_iv: "b6c5a8b1eae78b9fba789a547b197761"
server_key_version: "encryptionkey@0"
I20260927 05:03:48.090229 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "108d4327bf4a490e8dc78f356ea5ed40"
format_stamp: "Formatted at 2026-09-27 05:03:48 on dist-test-slave-39f2"
server_key: "f45fea1d69e11a2fab80f4d12cd37494"
server_key_iv: "b6c5a8b1eae78b9fba789a547b197761"
server_key_version: "encryptionkey@0"
I20260927 05:03:48.095386 5999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:03:48.098896 6014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:48.100240 5999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:03:48.100442 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "108d4327bf4a490e8dc78f356ea5ed40"
format_stamp: "Formatted at 2026-09-27 05:03:48 on dist-test-slave-39f2"
server_key: "f45fea1d69e11a2fab80f4d12cd37494"
server_key_iv: "b6c5a8b1eae78b9fba789a547b197761"
server_key_version: "encryptionkey@0"
I20260927 05:03:48.100641 5999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:48.116312 5999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:48.120302 5999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:48.120590 5999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:48.140069 5999 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:34193
I20260927 05:03:48.140091 6065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:34193 every 8 connection(s)
I20260927 05:03:48.141469 5999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:03:48.143959 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 5999
I20260927 05:03:48.144186 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:03:48.144871 21157 external_mini_cluster.cc:1428] Setting key de75c03743cb300581aadefb06f95ebe
I20260927 05:03:48.145550 6066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:48.159757 6066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40: Bootstrap starting.
Sep 27 05:03:48 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:48.163416 6066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:48.164475 6066 log.cc:824] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:48.167953 6066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40: No bootstrap required, opened a new log
I20260927 05:03:48.170744 6069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:48.146930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.20.169.126:34193)
0927 05:03:48.147709 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:48.147724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:48.147783 (+ 59us) server_negotiation.cc:407] Connection header received
0927 05:03:48.150287 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:48.150315 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:48.150846 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:48.151328 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:48.152670 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:48.154699 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:48.156191 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:48.157355 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:48.157528 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:48.162139 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:48.162147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:48.162161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:48.162200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:48.165980 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:48.166856 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:48.166863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:48.166869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:48.167050 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:48.167504 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:48.167508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:48.167510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:48.168102 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:48.168372 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:48.168786 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 05:03:48.169330 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":321,"threads_started":1}
I20260927 05:03:48.176846 6066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d4327bf4a490e8dc78f356ea5ed40" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34193 } }
I20260927 05:03:48.177279 6066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:48.177398 6066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108d4327bf4a490e8dc78f356ea5ed40, State: Initialized, Role: FOLLOWER
I20260927 05:03:48.177942 6066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d4327bf4a490e8dc78f356ea5ed40" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34193 } }
I20260927 05:03:48.178161 6066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:03:48.178301 6066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:03:48.178455 6066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:48.180914 6066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d4327bf4a490e8dc78f356ea5ed40" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34193 } }
I20260927 05:03:48.181422 6066 leader_election.cc:302] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 108d4327bf4a490e8dc78f356ea5ed40; no voters:
I20260927 05:03:48.181953 6066 leader_election.cc:288] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:03:48.182098 6071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:48.182842 6071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [term 1 LEADER]: Becoming Leader. State: Replica: 108d4327bf4a490e8dc78f356ea5ed40, State: Running, Role: LEADER
I20260927 05:03:48.183399 6071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d4327bf4a490e8dc78f356ea5ed40" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34193 } }
I20260927 05:03:48.184016 6066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:03:48.186177 6072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "108d4327bf4a490e8dc78f356ea5ed40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d4327bf4a490e8dc78f356ea5ed40" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34193 } } }
I20260927 05:03:48.186443 6073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 108d4327bf4a490e8dc78f356ea5ed40. Latest consensus state: current_term: 1 leader_uuid: "108d4327bf4a490e8dc78f356ea5ed40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d4327bf4a490e8dc78f356ea5ed40" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34193 } } }
I20260927 05:03:48.186712 6073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:48.186712 6072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40 [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:48.188463 6078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:03:48.193604 6078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:03:48.202672 6078 catalog_manager.cc:1347] Generated new cluster ID: 88b06e771dd842ae9fb5f1c4b803c1cb
I20260927 05:03:48.202785 6078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:03:48.212306 6078 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:03:48.213665 6078 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:03:48.223189 6078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40: Generated new TSK 0
I20260927 05:03:48.224063 6078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:03:48.300585 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34193
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:48.655380 6092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:48.655644 6092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:48.655810 6092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:48.662788 6092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:48.662891 6092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:48.663069 6092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:03:48.673203 6092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:48.674774 6092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:48.676218 6092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:48.690706 6097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:48.690898 6098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:48.691891 6100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:48.692461 6092 server_base.cc:1034] running on GCE node
I20260927 05:03:48.693121 6092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:48.694034 6092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:48.695286 6092 hybrid_clock.cc:648] HybridClock initialized: now 1790485428695177 us; error 78 us; skew 500 ppm
I20260927 05:03:48.695694 6092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:48 dist-test-slave-39f2 krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485428, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:48.699028 6092 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:03:48.700574 6092 webserver.cc:460] Webserver started at http://127.20.169.65:37999/ using document root <none> and password file <none>
I20260927 05:03:48.701361 6092 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:48.701470 6092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:48.701745 6092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:48.704998 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "f8441b9d4d8a449e8b7c1feced74b6a9"
format_stamp: "Formatted at 2026-09-27 05:03:48 on dist-test-slave-39f2"
server_key: "03299a79ec8e9da40dff3499d304b7dd"
server_key_iv: "725a1fb4e01ad5f14f62329d91503a77"
server_key_version: "encryptionkey@0"
I20260927 05:03:48.705848 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "f8441b9d4d8a449e8b7c1feced74b6a9"
format_stamp: "Formatted at 2026-09-27 05:03:48 on dist-test-slave-39f2"
server_key: "03299a79ec8e9da40dff3499d304b7dd"
server_key_iv: "725a1fb4e01ad5f14f62329d91503a77"
server_key_version: "encryptionkey@0"
I20260927 05:03:48.711304 6092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:03:48.715133 6107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:48.716537 6092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:03:48.716729 6092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "f8441b9d4d8a449e8b7c1feced74b6a9"
format_stamp: "Formatted at 2026-09-27 05:03:48 on dist-test-slave-39f2"
server_key: "03299a79ec8e9da40dff3499d304b7dd"
server_key_iv: "725a1fb4e01ad5f14f62329d91503a77"
server_key_version: "encryptionkey@0"
I20260927 05:03:48.717017 6092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:48.743126 6092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:48.747431 6092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:48.747781 6092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:48.748919 6092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:48.750309 6092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:48.750418 6092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:48.750586 6092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:48.750667 6092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:48.786798 6092 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:34631
I20260927 05:03:48.786846 6219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:34631 every 8 connection(s)
I20260927 05:03:48.788417 6092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:03:48.796926 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6092
I20260927 05:03:48.797156 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:03:48.797932 21157 external_mini_cluster.cc:1428] Setting key 2903b053c6a4b78e27d51eb3f92e9df7
Sep 27 05:03:48 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485428, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:48.812604 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:48.791292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:51943 (local address 127.20.169.126:34193)
0927 05:03:48.791736 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:48.791741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:48.792840 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 05:03:48.794887 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:48.794891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:48.794959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:48.795059 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:48.796865 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:48.797710 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:48.799943 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:48.800398 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:48.800555 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:48.804998 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:48.805008 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:48.805016 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:48.805066 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:48.809077 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:48.809905 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:48.809915 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:48.809922 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:48.810110 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:48.810726 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:48.810730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:48.810732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:48.811017 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:48.811133 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:48.812140 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 05:03:48.812329 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260927 05:03:48.814507 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:48.791881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34193 (local address 127.20.169.65:51943)
0927 05:03:48.792616 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:48.792660 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:48.794597 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:48.795284 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:48.795293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:48.795866 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:48.796605 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:48.796636 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:48.797885 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:48.797891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:48.799775 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:48.799792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:48.800647 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:48.800657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:48.800992 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:48.801794 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:48.801818 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:48.804646 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:48.809234 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:48.809242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:48.809301 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:48.809767 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:48.810249 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:48.810315 (+ 66us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:48.810322 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:48.810604 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:48.811179 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:48.811188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:48.811711 (+ 523us) client_negotiation.cc:769] Sending connection context
0927 05:03:48.812050 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 05:03:48.812695 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":273,"threads_started":1}
I20260927 05:03:48.816223 6220 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34193
I20260927 05:03:48.816550 6220 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:48.817373 6220 heartbeater.cc:511] Master 127.20.169.126:34193 requested a full tablet report, sending...
I20260927 05:03:48.819521 6031 ts_manager.cc:194] Registered new tserver with Master: f8441b9d4d8a449e8b7c1feced74b6a9 (127.20.169.65:34631)
I20260927 05:03:48.821151 6031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:51943
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:03:48.860417 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34193
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:49.199838 6226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:49.200114 6226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:49.200274 6226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:49.207633 6226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:49.207737 6226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:49.207922 6226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:03:49.217682 6226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:49.218961 6226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:49.220358 6226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:49.234375 6231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:49.234372 6234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:49.234421 6232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:49.235253 6226 server_base.cc:1034] running on GCE node
I20260927 05:03:49.235932 6226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:49.236917 6226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:49.238140 6226 hybrid_clock.cc:648] HybridClock initialized: now 1790485429238085 us; error 51 us; skew 500 ppm
I20260927 05:03:49.238474 6226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:49 dist-test-slave-39f2 krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:49.241595 6226 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:03:49.242981 6226 webserver.cc:460] Webserver started at http://127.20.169.66:38811/ using document root <none> and password file <none>
I20260927 05:03:49.243652 6226 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:49.243795 6226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:49.244107 6226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:49.247009 6226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "b8c2c1d926cc4aadb2383f6407e24b66"
format_stamp: "Formatted at 2026-09-27 05:03:49 on dist-test-slave-39f2"
server_key: "a3068d51c1f21c67ac21afaa27556a77"
server_key_iv: "610dbd2eed61c42f46d5eff53b3a5314"
server_key_version: "encryptionkey@0"
I20260927 05:03:49.247746 6226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "b8c2c1d926cc4aadb2383f6407e24b66"
format_stamp: "Formatted at 2026-09-27 05:03:49 on dist-test-slave-39f2"
server_key: "a3068d51c1f21c67ac21afaa27556a77"
server_key_iv: "610dbd2eed61c42f46d5eff53b3a5314"
server_key_version: "encryptionkey@0"
I20260927 05:03:49.252921 6226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 05:03:49.256585 6241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:49.258011 6226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:49.258191 6226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "b8c2c1d926cc4aadb2383f6407e24b66"
format_stamp: "Formatted at 2026-09-27 05:03:49 on dist-test-slave-39f2"
server_key: "a3068d51c1f21c67ac21afaa27556a77"
server_key_iv: "610dbd2eed61c42f46d5eff53b3a5314"
server_key_version: "encryptionkey@0"
I20260927 05:03:49.258407 6226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:49.273530 6226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:49.278292 6226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:49.278640 6226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:49.279626 6226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:49.281104 6226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:49.281215 6226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:49.281379 6226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:49.281474 6226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:49.316615 6226 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:44829
I20260927 05:03:49.316651 6353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:44829 every 8 connection(s)
I20260927 05:03:49.317939 6226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:03:49.325645 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6226
I20260927 05:03:49.325856 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:03:49.326315 21157 external_mini_cluster.cc:1428] Setting key 892ca77bebd8364d860b85800d7f405d
Sep 27 05:03:49 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:49.340564 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.320579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40079 (local address 127.20.169.126:34193)
0927 05:03:49.321085 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:49.321090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:49.322050 (+ 960us) server_negotiation.cc:407] Connection header received
0927 05:03:49.323912 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:49.323917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:49.324003 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:49.324137 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:49.325658 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.326538 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.328210 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.328965 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.329126 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.332888 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:49.332898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:49.332906 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:49.332957 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:49.337108 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:49.337931 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:49.337940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:49.337947 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:49.338129 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:49.338640 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:49.338648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:49.338655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:49.338955 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:49.339072 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:49.340071 (+ 999us) server_negotiation.cc:299] Negotiation successful
0927 05:03:49.340250 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":297,"threads_started":1}
I20260927 05:03:49.342108 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.321193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34193 (local address 127.20.169.66:40079)
0927 05:03:49.321888 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:49.321924 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:49.323653 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:49.324370 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:49.324379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:49.324938 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:49.325494 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.325512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.326685 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.326689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.328044 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.328057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.329179 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.329184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.329432 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.330158 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:49.330181 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:49.332607 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:49.337329 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:49.337336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:49.337348 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:49.337801 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:49.338296 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:49.338300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:49.338302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:49.338501 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:49.339093 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:49.339103 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:49.339608 (+ 505us) client_negotiation.cc:769] Sending connection context
0927 05:03:49.339988 (+ 380us) client_negotiation.cc:240] Negotiation successful
0927 05:03:49.340571 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":306,"threads_started":1}
I20260927 05:03:49.343719 6354 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34193
I20260927 05:03:49.344044 6354 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:49.344758 6354 heartbeater.cc:511] Master 127.20.169.126:34193 requested a full tablet report, sending...
I20260927 05:03:49.346346 6031 ts_manager.cc:194] Registered new tserver with Master: b8c2c1d926cc4aadb2383f6407e24b66 (127.20.169.66:44829)
I20260927 05:03:49.347178 6031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:40079
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:03:49.387916 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34193
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:49.719014 6360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:49.719363 6360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:49.719547 6360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:49.727031 6360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:49.727183 6360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:49.727382 6360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:03:49.737844 6360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:49.739179 6360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:49.740567 6360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:49.754266 6365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:49.754314 6368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:49.754348 6366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:49.755179 6360 server_base.cc:1034] running on GCE node
I20260927 05:03:49.755710 6360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:49.756642 6360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:49.757880 6360 hybrid_clock.cc:648] HybridClock initialized: now 1790485429757823 us; error 54 us; skew 500 ppm
I20260927 05:03:49.758189 6360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:49 dist-test-slave-39f2 krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:49.761250 6360 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:03:49.762689 6360 webserver.cc:460] Webserver started at http://127.20.169.67:36545/ using document root <none> and password file <none>
I20260927 05:03:49.763423 6360 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:49.763564 6360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:49.763880 6360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:49.766788 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "9c5e2b084c2c4374a95dad5cac6fb104"
format_stamp: "Formatted at 2026-09-27 05:03:49 on dist-test-slave-39f2"
server_key: "996af282ca1a2c594638e525b8615441"
server_key_iv: "6f7571dce0cfe85c5a5825bfed1120e2"
server_key_version: "encryptionkey@0"
I20260927 05:03:49.767606 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "9c5e2b084c2c4374a95dad5cac6fb104"
format_stamp: "Formatted at 2026-09-27 05:03:49 on dist-test-slave-39f2"
server_key: "996af282ca1a2c594638e525b8615441"
server_key_iv: "6f7571dce0cfe85c5a5825bfed1120e2"
server_key_version: "encryptionkey@0"
I20260927 05:03:49.772802 6360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 05:03:49.776260 6375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:49.777557 6360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:03:49.777752 6360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "9c5e2b084c2c4374a95dad5cac6fb104"
format_stamp: "Formatted at 2026-09-27 05:03:49 on dist-test-slave-39f2"
server_key: "996af282ca1a2c594638e525b8615441"
server_key_iv: "6f7571dce0cfe85c5a5825bfed1120e2"
server_key_version: "encryptionkey@0"
I20260927 05:03:49.777958 6360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:49.794668 6360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:49.798686 6360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:49.798981 6360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:49.799912 6360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:49.801280 6360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:49.801391 6360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:49.801550 6360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:49.801626 6360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:49.824416 6220 heartbeater.cc:503] Master 127.20.169.126:34193 was elected leader, sending a full tablet report...
I20260927 05:03:49.836263 6360 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:45025
I20260927 05:03:49.836321 6487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:45025 every 8 connection(s)
I20260927 05:03:49.837595 6360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:03:49.842267 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6360
I20260927 05:03:49.842471 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:03:49.843034 21157 external_mini_cluster.cc:1428] Setting key b340d8a8e03006736c12cf0f924b7e6b
Sep 27 05:03:49 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485429, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:49.859197 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.840253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:45729 (local address 127.20.169.126:34193)
0927 05:03:49.840444 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:49.840449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:49.841808 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 05:03:49.844244 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:49.844251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:49.844382 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:49.844515 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:49.846397 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.847222 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.848989 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.849446 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.849590 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.853257 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:49.853263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:49.853267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:49.853303 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:49.855926 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:49.856678 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:49.856688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:49.856691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:49.856896 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:49.857350 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:49.857356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:49.857359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:49.857643 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:49.857838 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:49.858562 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 05:03:49.858836 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:03:49.860253 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.840753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34193 (local address 127.20.169.67:45729)
0927 05:03:49.841605 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:49.841654 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:49.843943 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:49.844961 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:49.844970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:49.845569 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:49.846206 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.846224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.847394 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.847399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.848761 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.848776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.849689 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.849697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.849944 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.850642 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:49.850666 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:49.853037 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:49.856071 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:49.856080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:49.856097 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:49.856540 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:49.857011 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:49.857017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:49.857021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:49.857220 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:49.857794 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:49.857803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:49.858158 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 05:03:49.858437 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 05:03:49.858982 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":301,"threads_started":1}
I20260927 05:03:49.861748 6488 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34193
I20260927 05:03:49.862093 6488 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:49.862929 6488 heartbeater.cc:511] Master 127.20.169.126:34193 requested a full tablet report, sending...
I20260927 05:03:49.864519 6031 ts_manager.cc:194] Registered new tserver with Master: 9c5e2b084c2c4374a95dad5cac6fb104 (127.20.169.67:45025)
I20260927 05:03:49.865442 6031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:45729
I20260927 05:03:49.870808 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:03:49.890820 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.877143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.20.169.126:34193)
0927 05:03:49.877345 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:49.877351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:49.877368 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:03:49.877433 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:49.877436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:49.877493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:49.877609 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:49.879167 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.879963 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.881595 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.882141 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.882292 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.884947 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:49.884952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:49.884956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:49.884988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:49.887772 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:49.888482 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:49.888489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:49.888492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:49.888617 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:49.889113 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:49.889139 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:49.889143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:49.889482 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:49.889788 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:49.890178 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 05:03:49.890407 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:03:49.894824 6031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:03:49.901597 6031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:03:49.935539 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.914681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.20.169.67:45025)
0927 05:03:49.924857 (+ 10176us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:49.924869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:49.926148 (+ 1279us) server_negotiation.cc:407] Connection header received
0927 05:03:49.926531 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:49.926539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:49.926709 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:49.926919 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:49.928073 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.929832 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.932719 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.934237 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.934379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.934628 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:49.935001 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 05:03:49.935171 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9970,"thread_start_us":9807,"threads_started":1}
I20260927 05:03:49.935638 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.914962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34631 (local address 127.0.0.1:35344)
0927 05:03:49.925865 (+ 10903us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:49.925918 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:49.926112 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:49.927230 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:49.927252 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:49.927279 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:49.928206 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.928221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.930199 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.930208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.932673 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.932690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.934767 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.934771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.934928 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.934962 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:03:49.935052 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 05:03:49.935182 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10667,"mutex_wait_us":376,"spinlock_wait_cycles":30464,"thread_start_us":10533,"threads_started":1}
I20260927 05:03:49.936168 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.914584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:45025 (local address 127.0.0.1:33028)
0927 05:03:49.925866 (+ 11282us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:49.925916 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:49.926112 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:49.927230 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:49.927238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:49.927274 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:49.927766 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.927784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.929977 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.929983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.932564 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.932576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.934488 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.934498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.934709 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.934770 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 05:03:49.934937 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 05:03:49.935156 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10992,"mutex_wait_us":40,"spinlock_wait_cycles":8192,"thread_start_us":10776,"threads_started":1}
I20260927 05:03:49.936177 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.915555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.20.169.65:34631)
0927 05:03:49.925307 (+ 9752us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:49.925320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:49.926310 (+ 990us) server_negotiation.cc:407] Connection header received
0927 05:03:49.926437 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:49.926445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:49.926698 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:49.926920 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:49.928441 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.930051 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.932899 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.934247 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.934383 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.934626 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:49.935392 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 05:03:49.935583 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9545,"thread_start_us":9342,"threads_started":1}
I20260927 05:03:49.936412 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.926585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.20.169.66:44829)
0927 05:03:49.927889 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:49.927899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:49.928311 (+ 412us) server_negotiation.cc:407] Connection header received
0927 05:03:49.928512 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:49.928519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:49.928743 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:49.928984 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:49.929475 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.930865 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.933404 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.934471 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.934623 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.934833 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:49.935870 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 05:03:49.936049 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":997,"threads_started":1}
I20260927 05:03:49.936895 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:49.927472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:44829 (local address 127.0.0.1:54672)
0927 05:03:49.928176 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:49.928192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:49.928347 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:49.928969 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:49.928974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:49.929000 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:49.929326 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.929335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.931049 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.931055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.933248 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:49.933260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:49.934994 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:49.935001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:49.935179 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:49.935212 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:03:49.935308 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:03:49.935433 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":33,"thread_start_us":994,"threads_started":1}
I20260927 05:03:49.939703 6423 tablet_service.cc:1467] Processing CreateTablet for tablet af0933cb658b44d38249beb7651403ef (DEFAULT_TABLE table=test-table [id=c1c9afa3307a407fada27b226e075e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:49.940169 6155 tablet_service.cc:1467] Processing CreateTablet for tablet af0933cb658b44d38249beb7651403ef (DEFAULT_TABLE table=test-table [id=c1c9afa3307a407fada27b226e075e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:49.940157 6289 tablet_service.cc:1467] Processing CreateTablet for tablet af0933cb658b44d38249beb7651403ef (DEFAULT_TABLE table=test-table [id=c1c9afa3307a407fada27b226e075e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:49.941488 6423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0933cb658b44d38249beb7651403ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:49.941612 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0933cb658b44d38249beb7651403ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:49.941700 6155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0933cb658b44d38249beb7651403ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:49.952867 6506 tablet_bootstrap.cc:492] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Bootstrap starting.
I20260927 05:03:49.955953 6506 tablet_bootstrap.cc:654] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:49.957455 6506 log.cc:824] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:49.961674 6507 tablet_bootstrap.cc:492] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Bootstrap starting.
I20260927 05:03:49.962700 6508 tablet_bootstrap.cc:492] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Bootstrap starting.
I20260927 05:03:49.966154 6507 tablet_bootstrap.cc:654] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:49.966217 6508 tablet_bootstrap.cc:654] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:49.966308 6506 tablet_bootstrap.cc:492] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: No bootstrap required, opened a new log
I20260927 05:03:49.966583 6506 ts_tablet_manager.cc:1397] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 05:03:49.967543 6507 log.cc:824] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:49.968271 6508 log.cc:824] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:49.970788 6507 tablet_bootstrap.cc:492] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: No bootstrap required, opened a new log
I20260927 05:03:49.970817 6508 tablet_bootstrap.cc:492] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: No bootstrap required, opened a new log
I20260927 05:03:49.971047 6507 ts_tablet_manager.cc:1397] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 05:03:49.971081 6508 ts_tablet_manager.cc:1397] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 05:03:49.977268 6506 raft_consensus.cc:348] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:49.977694 6506 raft_consensus.cc:374] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:49.977792 6506 raft_consensus.cc:729] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5e2b084c2c4374a95dad5cac6fb104, State: Initialized, Role: FOLLOWER
I20260927 05:03:49.978403 6506 consensus_queue.cc:260] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:49.979059 6508 raft_consensus.cc:348] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:49.979472 6508 raft_consensus.cc:374] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:49.979547 6488 heartbeater.cc:503] Master 127.20.169.126:34193 was elected leader, sending a full tablet report...
I20260927 05:03:49.979584 6508 raft_consensus.cc:729] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2c1d926cc4aadb2383f6407e24b66, State: Initialized, Role: FOLLOWER
I20260927 05:03:49.980134 6506 ts_tablet_manager.cc:1428] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 05:03:49.980259 6508 consensus_queue.cc:260] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:49.981674 6507 raft_consensus.cc:348] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:49.982086 6507 raft_consensus.cc:374] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:49.982196 6507 raft_consensus.cc:729] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8441b9d4d8a449e8b7c1feced74b6a9, State: Initialized, Role: FOLLOWER
I20260927 05:03:49.982555 6354 heartbeater.cc:503] Master 127.20.169.126:34193 was elected leader, sending a full tablet report...
I20260927 05:03:49.982985 6508 ts_tablet_manager.cc:1428] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260927 05:03:49.982854 6507 consensus_queue.cc:260] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:49.985311 6507 ts_tablet_manager.cc:1428] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260927 05:03:50.041311 6221 tablet.cc:2369] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:03:50.069950 6355 tablet.cc:2369] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:03:50.089346 6489 tablet.cc:2369] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:03:50.171514 6512 raft_consensus.cc:482] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:50.171964 6512 raft_consensus.cc:504] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:50.174103 6512 leader_election.cc:288] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8c2c1d926cc4aadb2383f6407e24b66 (127.20.169.66:44829), f8441b9d4d8a449e8b7c1feced74b6a9 (127.20.169.65:34631)
I20260927 05:03:50.179140 6513 raft_consensus.cc:482] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:50.179445 6513 raft_consensus.cc:504] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:50.181409 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.174562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:50947 (local address 127.20.169.65:34631)
0927 05:03:50.174791 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.174800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.174880 (+ 80us) server_negotiation.cc:407] Connection header received
0927 05:03:50.175255 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.175269 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.175384 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.175496 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:50.176038 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.177222 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.179133 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.180345 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.180519 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.180601 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.180981 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.181122 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:03:50.181716 6513 leader_election.cc:288] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c5e2b084c2c4374a95dad5cac6fb104 (127.20.169.67:45025), f8441b9d4d8a449e8b7c1feced74b6a9 (127.20.169.65:34631)
I20260927 05:03:50.182029 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.174534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34631 (local address 127.20.169.67:50947)
0927 05:03:50.174755 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:50.174778 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:50.174979 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:50.175528 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:50.175531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:50.175556 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:50.175819 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.175830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.177374 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.177380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.178965 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.178981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.180520 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.180529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.180737 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.180786 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 05:03:50.180918 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:03:50.181070 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1,"spinlock_wait_cycles":48000}
I20260927 05:03:50.182622 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.174525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:49047 (local address 127.20.169.66:44829)
0927 05:03:50.175010 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.175019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.175634 (+ 615us) server_negotiation.cc:407] Connection header received
0927 05:03:50.175824 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.175829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.175922 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.176012 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:50.176685 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.177505 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.179664 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.180956 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.181101 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.181164 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.181735 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.181944 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260927 05:03:50.182710 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.174547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:44829 (local address 127.20.169.67:49047)
0927 05:03:50.175517 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:50.175536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:50.175686 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:50.176063 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:50.176068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:50.176098 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:50.176490 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.176506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.177662 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.177666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.179471 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.179487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.181339 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.181348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.181537 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.181578 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 05:03:50.182181 (+ 603us) client_negotiation.cc:240] Negotiation successful
0927 05:03:50.182317 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":101,"thread_start_us":456,"threads_started":1}
I20260927 05:03:50.183918 6175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0933cb658b44d38249beb7651403ef" candidate_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" is_pre_election: true
I20260927 05:03:50.184526 6175 raft_consensus.cc:2454] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5e2b084c2c4374a95dad5cac6fb104 in term 0.
I20260927 05:03:50.184463 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0933cb658b44d38249beb7651403ef" candidate_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" is_pre_election: true
I20260927 05:03:50.185030 6309 raft_consensus.cc:2454] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5e2b084c2c4374a95dad5cac6fb104 in term 0.
I20260927 05:03:50.185322 6379 leader_election.cc:302] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c5e2b084c2c4374a95dad5cac6fb104, f8441b9d4d8a449e8b7c1feced74b6a9; no voters:
I20260927 05:03:50.185753 6512 raft_consensus.cc:2781] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:03:50.185922 6512 raft_consensus.cc:482] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:03:50.186044 6512 raft_consensus.cc:3037] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:50.189109 6512 raft_consensus.cc:504] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:50.190191 6512 leader_election.cc:288] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [CANDIDATE]: Term 1 election: Requested vote from peers b8c2c1d926cc4aadb2383f6407e24b66 (127.20.169.66:44829), f8441b9d4d8a449e8b7c1feced74b6a9 (127.20.169.65:34631)
I20260927 05:03:50.190371 6175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0933cb658b44d38249beb7651403ef" candidate_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9"
I20260927 05:03:50.190534 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0933cb658b44d38249beb7651403ef" candidate_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2c1d926cc4aadb2383f6407e24b66"
I20260927 05:03:50.190568 6175 raft_consensus.cc:3037] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:50.190719 6309 raft_consensus.cc:3037] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:50.191236 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.182198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:34631 (local address 127.20.169.66:41765)
0927 05:03:50.183754 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:50.183778 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:50.183961 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:50.184502 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:50.184507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:50.184536 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:50.185031 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.185045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.186463 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.186469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.188296 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.188308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.190258 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.190266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.190467 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.190504 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:03:50.190609 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 05:03:50.190749 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"mutex_wait_us":6,"thread_start_us":1271,"threads_started":1}
I20260927 05:03:50.191982 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.182032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:41765 (local address 127.20.169.65:34631)
0927 05:03:50.182895 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.182899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.183857 (+ 958us) server_negotiation.cc:407] Connection header received
0927 05:03:50.184129 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.184134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.184208 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.184270 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:50.185204 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.186339 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.188506 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.189804 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.189985 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.190087 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.191497 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.191660 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":370,"server-negotiator.queue_time_us":758}
I20260927 05:03:50.192538 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.182100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:54943 (local address 127.20.169.67:45025)
0927 05:03:50.182325 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.182334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.184087 (+ 1753us) server_negotiation.cc:407] Connection header received
0927 05:03:50.184366 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.184374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.184448 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.184517 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:50.185780 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.187107 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.189636 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.191249 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.191442 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.191519 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.192138 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.192283 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:03:50.192523 6174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0933cb658b44d38249beb7651403ef" candidate_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" is_pre_election: true
I20260927 05:03:50.193303 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.182634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:45025 (local address 127.20.169.66:54943)
0927 05:03:50.183987 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:50.184006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:50.184199 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:50.184856 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:50.184868 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:50.184899 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:50.185569 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.185587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.187262 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.187268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.189416 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:50.189430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.191689 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.191696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:50.191906 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.191951 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 05:03:50.192745 (+ 794us) client_negotiation.cc:240] Negotiation successful
0927 05:03:50.192908 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":59,"thread_start_us":400,"threads_started":1}
I20260927 05:03:50.193964 6175 raft_consensus.cc:2454] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5e2b084c2c4374a95dad5cac6fb104 in term 1.
I20260927 05:03:50.194301 6309 raft_consensus.cc:2454] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5e2b084c2c4374a95dad5cac6fb104 in term 1.
I20260927 05:03:50.194103 6443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0933cb658b44d38249beb7651403ef" candidate_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" is_pre_election: true
I20260927 05:03:50.194448 6379 leader_election.cc:302] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c5e2b084c2c4374a95dad5cac6fb104, f8441b9d4d8a449e8b7c1feced74b6a9; no voters:
I20260927 05:03:50.194743 6443 raft_consensus.cc:2379] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8c2c1d926cc4aadb2383f6407e24b66 in current term 1: Already voted for candidate 9c5e2b084c2c4374a95dad5cac6fb104 in this term.
I20260927 05:03:50.195015 6512 raft_consensus.cc:2781] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:50.195515 6512 raft_consensus.cc:686] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 1 LEADER]: Becoming Leader. State: Replica: 9c5e2b084c2c4374a95dad5cac6fb104, State: Running, Role: LEADER
I20260927 05:03:50.195601 6244 leader_election.cc:302] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8c2c1d926cc4aadb2383f6407e24b66; no voters: 9c5e2b084c2c4374a95dad5cac6fb104, f8441b9d4d8a449e8b7c1feced74b6a9
I20260927 05:03:50.195988 6513 raft_consensus.cc:2726] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 05:03:50.195992 6512 consensus_queue.cc:237] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } }
I20260927 05:03:50.200455 6029 catalog_manager.cc:5495] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c5e2b084c2c4374a95dad5cac6fb104 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5e2b084c2c4374a95dad5cac6fb104" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 } health_report { overall_health: UNKNOWN } } }
I20260927 05:03:50.224611 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.218767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.20.169.67:45025)
0927 05:03:50.219048 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.219055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.219137 (+ 82us) server_negotiation.cc:407] Connection header received
0927 05:03:50.219453 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.219457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.219552 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.219704 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:03:50.220114 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.220956 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.222453 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.222859 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.222995 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.223490 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:03:50.223999 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:03:50.224161 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.224241 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.224320 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 05:03:50.232944 6175 raft_consensus.cc:1261] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 1 FOLLOWER]: Refusing update from remote peer 9c5e2b084c2c4374a95dad5cac6fb104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:50.233012 6309 raft_consensus.cc:1261] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 1 FOLLOWER]: Refusing update from remote peer 9c5e2b084c2c4374a95dad5cac6fb104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:50.233976 6518 consensus_queue.cc:1035] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8441b9d4d8a449e8b7c1feced74b6a9" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:50.234352 6512 consensus_queue.cc:1035] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8c2c1d926cc4aadb2383f6407e24b66" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:50.259737 6031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:50.260041 6031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:50.263792 6031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 108d4327bf4a490e8dc78f356ea5ed40: Sending DeleteTablet for 3 replicas of tablet af0933cb658b44d38249beb7651403ef
I20260927 05:03:50.264854 6423 tablet_service.cc:1514] Processing DeleteTablet for tablet af0933cb658b44d38249beb7651403ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:50 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:33028
I20260927 05:03:50.265050 6289 tablet_service.cc:1514] Processing DeleteTablet for tablet af0933cb658b44d38249beb7651403ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:50 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:54672
I20260927 05:03:50.265143 6155 tablet_service.cc:1514] Processing DeleteTablet for tablet af0933cb658b44d38249beb7651403ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:50 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:35344
I20260927 05:03:50.269380 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6092
I20260927 05:03:50.277330 6532 tablet_replica.cc:330] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: stopping tablet replica
I20260927 05:03:50.277900 6532 raft_consensus.cc:2229] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:50.278542 6532 raft_consensus.cc:2258] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:50.279141 6531 tablet_replica.cc:330] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: stopping tablet replica
I20260927 05:03:50.279444 6533 tablet_replica.cc:330] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: stopping tablet replica
I20260927 05:03:50.279893 6533 raft_consensus.cc:2229] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:50.280085 6531 raft_consensus.cc:2229] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:03:50.280418 6533 raft_consensus.cc:2258] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:50.280756 6531 raft_consensus.cc:2258] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:50.281982 6532 ts_tablet_manager.cc:1905] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:50.283047 6533 ts_tablet_manager.cc:1905] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:50.283164 6531 ts_tablet_manager.cc:1905] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:50.289881 6532 ts_tablet_manager.cc:1918] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:50.290128 6532 log.cc:1196] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/af0933cb658b44d38249beb7651403ef
I20260927 05:03:50.290822 6532 ts_tablet_manager.cc:1939] T af0933cb658b44d38249beb7651403ef P b8c2c1d926cc4aadb2383f6407e24b66: Deleting consensus metadata
I20260927 05:03:50.290867 6533 ts_tablet_manager.cc:1918] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:50.291033 6533 log.cc:1196] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/af0933cb658b44d38249beb7651403ef
I20260927 05:03:50.291551 6533 ts_tablet_manager.cc:1939] T af0933cb658b44d38249beb7651403ef P f8441b9d4d8a449e8b7c1feced74b6a9: Deleting consensus metadata
I20260927 05:03:50.292969 6017 catalog_manager.cc:4841] TS b8c2c1d926cc4aadb2383f6407e24b66 (127.20.169.66:44829): tablet af0933cb658b44d38249beb7651403ef (table test-table [id=c1c9afa3307a407fada27b226e075e91]) successfully deleted
I20260927 05:03:50.293435 6018 catalog_manager.cc:4841] TS f8441b9d4d8a449e8b7c1feced74b6a9 (127.20.169.65:34631): tablet af0933cb658b44d38249beb7651403ef (table test-table [id=c1c9afa3307a407fada27b226e075e91]) successfully deleted
Sep 27 05:03:50 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:03:50.296028 6531 ts_tablet_manager.cc:1918] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:50.296224 6531 log.cc:1196] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/af0933cb658b44d38249beb7651403ef
I20260927 05:03:50.296717 6531 ts_tablet_manager.cc:1939] T af0933cb658b44d38249beb7651403ef P 9c5e2b084c2c4374a95dad5cac6fb104: Deleting consensus metadata
I20260927 05:03:50.298448 6017 catalog_manager.cc:4841] TS 9c5e2b084c2c4374a95dad5cac6fb104 (127.20.169.67:45025): tablet af0933cb658b44d38249beb7651403ef (table test-table [id=c1c9afa3307a407fada27b226e075e91]) successfully deleted
I20260927 05:03:50.303550 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.270816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.20.169.65:34631)
0927 05:03:50.271005 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.271012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.276412 (+ 5400us) server_negotiation.cc:407] Connection header received
0927 05:03:50.276633 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.276639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.276735 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.276915 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:50.278929 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.280307 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.284975 (+ 4668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.285782 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.285894 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.297141 (+ 11247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:50.297157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:50.297175 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:50.297217 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:50.300572 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:50.301495 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:50.301502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:50.301505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:50.301633 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:50.302062 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:50.302066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:50.302068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:50.302404 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:50.302723 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.303029 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.303203 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:03:50.304335 6215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:50.396845 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6092
I20260927 05:03:50.424744 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6226
Sep 27 05:03:50 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:03:50.443291 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.425577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.20.169.66:44829)
0927 05:03:50.425761 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.425766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.425877 (+ 111us) server_negotiation.cc:407] Connection header received
0927 05:03:50.426064 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.426068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.426133 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.426218 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:50.428028 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.428755 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.430404 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.431126 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.431270 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.435731 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:50.435751 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:50.435781 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:50.435833 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:50.440175 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:50.440895 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:50.440906 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:50.440910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:50.441086 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:50.441500 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:50.441506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:50.441510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:50.442048 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:50.442358 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.442619 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.442880 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:03:50.444046 6349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:50.536480 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6226
I20260927 05:03:50.563800 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6360
Sep 27 05:03:50 dist-test-slave-39f2 krb5kdc[5986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:03:50.584228 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:50.564628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.20.169.67:45025)
0927 05:03:50.564889 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:50.564896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:50.565195 (+ 299us) server_negotiation.cc:407] Connection header received
0927 05:03:50.565396 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:50.565401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:50.565474 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:50.565567 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:50.567658 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.568708 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.570508 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:50.571038 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:50.571231 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:50.576917 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:50.576931 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:50.576948 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:50.576996 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:50.580668 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:50.581591 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:50.581599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:50.581601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:50.581763 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:50.582249 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:50.582261 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:50.582263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:50.582736 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:50.583099 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:50.583535 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 05:03:50.583838 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:03:50.585127 6483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:50.708009 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6360
I20260927 05:03:50.724977 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 5999
I20260927 05:03:50.726099 6061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:50.836547 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 5999
2026-09-27T05:03:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4516 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T05:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:03:50Z Disabled control of system clock
I20260927 05:03:50.919158 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:36519
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:36519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:03:51.247527 6542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:51.247792 6542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:51.247925 6542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:51.255141 6542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:03:51.255249 6542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:51.255287 6542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:51.255398 6542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:03:51.255518 6542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:03:51.264870 6542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:36519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:36519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:51.266204 6542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:51.267993 6542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:51.279706 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:51.279841 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:51.280051 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:51.281405 6542 server_base.cc:1034] running on GCE node
I20260927 05:03:51.282363 6542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:51.283901 6542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:51.285161 6542 hybrid_clock.cc:648] HybridClock initialized: now 1790485431285127 us; error 43 us; skew 500 ppm
I20260927 05:03:51.285538 6542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:51.291698 6542 webserver.cc:460] Webserver started at http://127.20.169.126:35531/ using document root <none> and password file <none>
I20260927 05:03:51.292531 6542 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:51.292662 6542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:51.293058 6542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:51.296064 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "7d17ec8813184341ace6fce679c30c85"
format_stamp: "Formatted at 2026-09-27 05:03:51 on dist-test-slave-39f2"
server_key: "23486549baab7ce27497928798969e29"
server_key_iv: "7e5cb103373ee10701d060f3cf76dedb"
server_key_version: "encryptionkey@0"
I20260927 05:03:51.296948 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "7d17ec8813184341ace6fce679c30c85"
format_stamp: "Formatted at 2026-09-27 05:03:51 on dist-test-slave-39f2"
server_key: "23486549baab7ce27497928798969e29"
server_key_iv: "7e5cb103373ee10701d060f3cf76dedb"
server_key_version: "encryptionkey@0"
I20260927 05:03:51.302078 6542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 05:03:51.305583 6556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:51.306958 6542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:03:51.307152 6542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "7d17ec8813184341ace6fce679c30c85"
format_stamp: "Formatted at 2026-09-27 05:03:51 on dist-test-slave-39f2"
server_key: "23486549baab7ce27497928798969e29"
server_key_iv: "7e5cb103373ee10701d060f3cf76dedb"
server_key_version: "encryptionkey@0"
I20260927 05:03:51.307380 6542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:51.325044 6542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:51.325894 6542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:51.326188 6542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:51.345350 6542 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:36519
I20260927 05:03:51.345445 6607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:36519 every 8 connection(s)
I20260927 05:03:51.346704 6542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:03:51.350800 6608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:51.352141 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6542
I20260927 05:03:51.352355 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:03:51.353034 21157 external_mini_cluster.cc:1428] Setting key 09624f63908156c85ebdb8adb2bcb403
I20260927 05:03:51.365170 6608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85: Bootstrap starting.
I20260927 05:03:51.367609 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:51.355156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.20.169.126:36519)
0927 05:03:51.357431 (+ 2275us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:51.357441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:51.357481 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:03:51.359069 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:51.359090 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:51.359532 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:51.359916 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:51.360469 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:51.361750 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:51.363172 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:51.363830 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:51.364025 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:51.365372 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:51.365383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:51.365403 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:51.365941 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:51.366015 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:51.366035 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:51.366166 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:51.366385 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 05:03:51.366683 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1856,"thread_start_us":1592,"threads_started":1}
I20260927 05:03:51.368649 6608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:51.369814 6608 log.cc:824] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:51.372916 6608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85: No bootstrap required, opened a new log
I20260927 05:03:51.380796 6608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17ec8813184341ace6fce679c30c85" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36519 } }
I20260927 05:03:51.381292 6608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:51.381461 6608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d17ec8813184341ace6fce679c30c85, State: Initialized, Role: FOLLOWER
I20260927 05:03:51.382194 6608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17ec8813184341ace6fce679c30c85" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36519 } }
I20260927 05:03:51.382471 6608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:03:51.382629 6608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:03:51.382831 6608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:51.385301 6608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17ec8813184341ace6fce679c30c85" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36519 } }
I20260927 05:03:51.385851 6608 leader_election.cc:302] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d17ec8813184341ace6fce679c30c85; no voters:
I20260927 05:03:51.386476 6608 leader_election.cc:288] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:03:51.386677 6613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:51.387522 6613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [term 1 LEADER]: Becoming Leader. State: Replica: 7d17ec8813184341ace6fce679c30c85, State: Running, Role: LEADER
I20260927 05:03:51.388090 6613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17ec8813184341ace6fce679c30c85" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36519 } }
I20260927 05:03:51.388856 6608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:03:51.391247 6615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d17ec8813184341ace6fce679c30c85. Latest consensus state: current_term: 1 leader_uuid: "7d17ec8813184341ace6fce679c30c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17ec8813184341ace6fce679c30c85" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36519 } } }
I20260927 05:03:51.392408 6615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:51.392216 6614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d17ec8813184341ace6fce679c30c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17ec8813184341ace6fce679c30c85" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 36519 } } }
I20260927 05:03:51.392930 6614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85 [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:51.398411 6624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:03:51.402974 6624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:03:51.411355 6624 catalog_manager.cc:1347] Generated new cluster ID: fd9250b385e34a64afa01638f58475e7
I20260927 05:03:51.411583 6624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:03:51.426299 6624 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:03:51.427683 6624 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:03:51.436897 6624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85: Generated new TSK 0
I20260927 05:03:51.437774 6624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:03:51.447405 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36519
--builtin_ntp_servers=127.20.169.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:03:51.788022 6632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:51.788281 6632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:51.788373 6632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:51.796059 6632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:51.796150 6632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:51.796250 6632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:03:51.805259 6632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:51.806438 6632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:51.808069 6632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:51.821058 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:51.821148 6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:51.821110 6637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:51.822161 6632 server_base.cc:1034] running on GCE node
I20260927 05:03:51.822755 6632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:51.823652 6632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:51.824945 6632 hybrid_clock.cc:648] HybridClock initialized: now 1790485431824915 us; error 60 us; skew 500 ppm
I20260927 05:03:51.825302 6632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:51.827529 6632 webserver.cc:460] Webserver started at http://127.20.169.65:37977/ using document root <none> and password file <none>
I20260927 05:03:51.828209 6632 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:51.828327 6632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:51.828647 6632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:51.831501 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "6adeea7d6fc64f9db478bee5b085a6d6"
format_stamp: "Formatted at 2026-09-27 05:03:51 on dist-test-slave-39f2"
server_key: "8cb46b680566fed0f112302a3ae2b7b2"
server_key_iv: "1b47aae6966e7dd418781bb07cae75f1"
server_key_version: "encryptionkey@0"
I20260927 05:03:51.832278 6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "6adeea7d6fc64f9db478bee5b085a6d6"
format_stamp: "Formatted at 2026-09-27 05:03:51 on dist-test-slave-39f2"
server_key: "8cb46b680566fed0f112302a3ae2b7b2"
server_key_iv: "1b47aae6966e7dd418781bb07cae75f1"
server_key_version: "encryptionkey@0"
I20260927 05:03:51.837321 6632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 05:03:51.840677 6646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:51.842100 6632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:03:51.842279 6632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "6adeea7d6fc64f9db478bee5b085a6d6"
format_stamp: "Formatted at 2026-09-27 05:03:51 on dist-test-slave-39f2"
server_key: "8cb46b680566fed0f112302a3ae2b7b2"
server_key_iv: "1b47aae6966e7dd418781bb07cae75f1"
server_key_version: "encryptionkey@0"
I20260927 05:03:51.842502 6632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:51.858752 6632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:51.859539 6632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:51.859836 6632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:51.860782 6632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:51.862273 6632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:51.862380 6632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:51.862509 6632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:51.862636 6632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:51.903977 6632 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:36061
I20260927 05:03:51.904066 6758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:36061 every 8 connection(s)
I20260927 05:03:51.905441 6632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:03:51.913352 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6632
I20260927 05:03:51.913542 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:03:51.914055 21157 external_mini_cluster.cc:1428] Setting key a69e41422f4cd4fadb381a0010c89d98
I20260927 05:03:51.917133 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36519
--builtin_ntp_servers=127.20.169.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:03:51.919201 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:51.908053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:54041 (local address 127.20.169.126:36519)
0927 05:03:51.908509 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:51.908514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:51.909882 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 05:03:51.911976 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:51.911982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:51.912095 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:51.912246 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:51.913687 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:51.914488 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:51.916257 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:51.916648 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:51.916747 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:51.918441 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:51.918446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:51.918449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:51.918466 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:51.918506 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:51.918517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:51.918588 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:51.918879 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 05:03:51.918955 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260927 05:03:51.920789 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:51.908688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36519 (local address 127.20.169.65:54041)
0927 05:03:51.909702 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:51.909741 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:51.911692 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:51.912502 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:51.912511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:51.912604 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:51.913484 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:51.913513 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:51.914658 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:51.914662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:51.916112 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:51.916128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:51.916932 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:51.916942 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:51.917272 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:51.918060 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:03:51.918067 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:51.918126 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:03:51.918235 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:51.918624 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:51.918640 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:51.918652 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:03:51.918848 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 05:03:51.919293 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":285,"threads_started":1}
I20260927 05:03:51.922387 6759 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36519
I20260927 05:03:51.922688 6759 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:51.923478 6759 heartbeater.cc:511] Master 127.20.169.126:36519 requested a full tablet report, sending...
I20260927 05:03:51.925659 6573 ts_manager.cc:194] Registered new tserver with Master: 6adeea7d6fc64f9db478bee5b085a6d6 (127.20.169.65:36061)
I20260927 05:03:51.927675 6573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.65:54041
W20260927 05:03:52.256428 6763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:52.256713 6763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:52.256882 6763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:52.264393 6763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:52.264498 6763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:52.264611 6763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:03:52.273877 6763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:52.275169 6763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:52.277006 6763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:52.290042 6769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:52.290104 6771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:52.290169 6768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:52.291231 6763 server_base.cc:1034] running on GCE node
I20260927 05:03:52.291957 6763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:52.293152 6763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:52.294384 6763 hybrid_clock.cc:648] HybridClock initialized: now 1790485432294325 us; error 46 us; skew 500 ppm
I20260927 05:03:52.294854 6763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:52.297459 6763 webserver.cc:460] Webserver started at http://127.20.169.66:37859/ using document root <none> and password file <none>
I20260927 05:03:52.298156 6763 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:52.298283 6763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:52.298630 6763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:52.301616 6763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a"
format_stamp: "Formatted at 2026-09-27 05:03:52 on dist-test-slave-39f2"
server_key: "16b86f438c7d705d7f924c671758e7a1"
server_key_iv: "6abc52db4d78ea87650660fe0c5be24e"
server_key_version: "encryptionkey@0"
I20260927 05:03:52.302455 6763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a"
format_stamp: "Formatted at 2026-09-27 05:03:52 on dist-test-slave-39f2"
server_key: "16b86f438c7d705d7f924c671758e7a1"
server_key_iv: "6abc52db4d78ea87650660fe0c5be24e"
server_key_version: "encryptionkey@0"
I20260927 05:03:52.307790 6763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:03:52.311275 6777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:52.312708 6763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:03:52.312922 6763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a"
format_stamp: "Formatted at 2026-09-27 05:03:52 on dist-test-slave-39f2"
server_key: "16b86f438c7d705d7f924c671758e7a1"
server_key_iv: "6abc52db4d78ea87650660fe0c5be24e"
server_key_version: "encryptionkey@0"
I20260927 05:03:52.313145 6763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:52.329905 6763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:52.330717 6763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:52.331051 6763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:52.332032 6763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:52.333565 6763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:52.333663 6763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:52.333812 6763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:52.333907 6763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:52.370640 6763 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:38593
I20260927 05:03:52.370702 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:38593 every 8 connection(s)
I20260927 05:03:52.372046 6763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:03:52.376282 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6763
I20260927 05:03:52.376467 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:03:52.377125 21157 external_mini_cluster.cc:1428] Setting key 3c924569a6575a7755b8664d3d72cd8b
I20260927 05:03:52.380303 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36519
--builtin_ntp_servers=127.20.169.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:03:52.387002 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.374837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:52997 (local address 127.20.169.126:36519)
0927 05:03:52.375019 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:52.375023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:52.376661 (+ 1638us) server_negotiation.cc:407] Connection header received
0927 05:03:52.378987 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:52.378996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:52.379089 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:52.379175 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:52.380317 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.381677 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.383385 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.384156 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.384322 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.385988 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:52.385999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:52.386009 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:52.386036 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:52.386082 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:52.386102 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:52.386204 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:52.386439 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 05:03:52.386557 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:03:52.387884 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.375350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36519 (local address 127.20.169.66:52997)
0927 05:03:52.376468 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:52.376531 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:52.378658 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:52.379442 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:52.379450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:52.379539 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:52.380127 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.380144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.381867 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.381870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.383214 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.383226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.384426 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.384440 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.384734 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.385624 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:03:52.385629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:52.385676 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:03:52.385765 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:52.386208 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:52.386219 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:52.386229 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:03:52.386392 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 05:03:52.386761 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":282,"threads_started":1}
I20260927 05:03:52.389516 6890 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36519
I20260927 05:03:52.389873 6890 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:52.390769 6890 heartbeater.cc:511] Master 127.20.169.126:36519 requested a full tablet report, sending...
I20260927 05:03:52.392643 6573 ts_manager.cc:194] Registered new tserver with Master: 61f0fd4e46fd41e78b7ceb4212d5de9a (127.20.169.66:38593)
I20260927 05:03:52.393595 6573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.66:52997
W20260927 05:03:52.725219 6893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:52.725558 6893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:52.725762 6893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:52.733155 6893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:52.733299 6893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:52.733534 6893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:03:52.742765 6893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:36519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:52.744102 6893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:52.745958 6893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:52.759462 6901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:52.759539 6898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:52.759537 6899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:52.760381 6893 server_base.cc:1034] running on GCE node
I20260927 05:03:52.761055 6893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:52.761963 6893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:52.763168 6893 hybrid_clock.cc:648] HybridClock initialized: now 1790485432763114 us; error 55 us; skew 500 ppm
I20260927 05:03:52.763494 6893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:52.765935 6893 webserver.cc:460] Webserver started at http://127.20.169.67:36367/ using document root <none> and password file <none>
I20260927 05:03:52.766651 6893 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:52.766777 6893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:52.767120 6893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:52.770153 6893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "21adfdff5ab74399b1a96b25b7a112b1"
format_stamp: "Formatted at 2026-09-27 05:03:52 on dist-test-slave-39f2"
server_key: "4f37e6c847e1145de915de3228f64607"
server_key_iv: "764f3a1ee6eeabf25e2cccc50a33cafe"
server_key_version: "encryptionkey@0"
I20260927 05:03:52.771019 6893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "21adfdff5ab74399b1a96b25b7a112b1"
format_stamp: "Formatted at 2026-09-27 05:03:52 on dist-test-slave-39f2"
server_key: "4f37e6c847e1145de915de3228f64607"
server_key_iv: "764f3a1ee6eeabf25e2cccc50a33cafe"
server_key_version: "encryptionkey@0"
I20260927 05:03:52.776278 6893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 05:03:52.779894 6907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:52.781378 6893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 05:03:52.781559 6893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "21adfdff5ab74399b1a96b25b7a112b1"
format_stamp: "Formatted at 2026-09-27 05:03:52 on dist-test-slave-39f2"
server_key: "4f37e6c847e1145de915de3228f64607"
server_key_iv: "764f3a1ee6eeabf25e2cccc50a33cafe"
server_key_version: "encryptionkey@0"
I20260927 05:03:52.781803 6893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:52.802692 6893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:52.803491 6893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:52.803804 6893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:52.804768 6893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:52.806226 6893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:52.806316 6893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:52.806433 6893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:52.806515 6893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:52.840495 6893 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:34123
I20260927 05:03:52.840586 7019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:34123 every 8 connection(s)
I20260927 05:03:52.841852 6893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:03:52.850029 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 6893
I20260927 05:03:52.850224 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:03:52.850677 21157 external_mini_cluster.cc:1428] Setting key 651dcce26dcb3e77c33ff41802dc6c2d
I20260927 05:03:52.855854 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.844595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:49253 (local address 127.20.169.126:36519)
0927 05:03:52.844777 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:52.844781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:52.846373 (+ 1592us) server_negotiation.cc:407] Connection header received
0927 05:03:52.848337 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:52.848344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:52.848443 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:52.848563 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:52.849764 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.850675 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.852758 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.853329 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.853480 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.854913 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:52.854927 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:52.854931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:52.854949 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:52.854984 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:52.854997 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:52.855102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:52.855364 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 05:03:52.855474 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:03:52.856909 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.845127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:36519 (local address 127.20.169.67:49253)
0927 05:03:52.846133 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:52.846170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:52.848027 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:52.848798 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:52.848847 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:52.848947 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:52.849551 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.849568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.850883 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.850888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.852585 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.852604 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.853564 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.853572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.853828 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.854578 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:03:52.854581 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:52.854620 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:03:52.854700 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:52.855101 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:52.855114 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:52.855126 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:03:52.855350 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 05:03:52.855743 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":383,"threads_started":1}
I20260927 05:03:52.858376 7020 heartbeater.cc:347] Connected to a master server at 127.20.169.126:36519
I20260927 05:03:52.858649 7020 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:52.859382 7020 heartbeater.cc:511] Master 127.20.169.126:36519 requested a full tablet report, sending...
I20260927 05:03:52.860895 6573 ts_manager.cc:194] Registered new tserver with Master: 21adfdff5ab74399b1a96b25b7a112b1 (127.20.169.67:34123)
I20260927 05:03:52.861716 6573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.67:49253
I20260927 05:03:52.865962 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:03:52.876075 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.870019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.20.169.126:36519)
0927 05:03:52.870252 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:52.870259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:52.870628 (+ 369us) server_negotiation.cc:407] Connection header received
0927 05:03:52.870861 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:52.870867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:52.870948 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:52.871053 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:52.871551 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.872267 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.873866 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.874273 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.874386 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.875501 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:52.875506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:52.875510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:52.875526 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:52.875556 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:52.875568 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:52.875643 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:52.875792 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 05:03:52.875864 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:03:52.879805 6573 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:03:52.886721 6573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:03:52.909509 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.898917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:38593 (local address 127.0.0.1:58678)
0927 05:03:52.899659 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:52.899710 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:52.899925 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:52.900900 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:52.900910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:52.900945 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:52.901244 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.901252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.903442 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.903447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.905973 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.905990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.908474 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.908483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.908688 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.908741 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 05:03:52.908894 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 05:03:52.909117 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
I20260927 05:03:52.911345 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.899112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.20.169.66:38593)
0927 05:03:52.899713 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:52.899722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:52.899796 (+ 74us) server_negotiation.cc:407] Connection header received
0927 05:03:52.900182 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:52.900194 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:52.900523 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:52.900884 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:52.901421 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.903312 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.906223 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.907907 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.908049 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.908315 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:52.910699 (+ 2384us) server_negotiation.cc:299] Negotiation successful
0927 05:03:52.910891 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
I20260927 05:03:52.913852 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.899972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56492)
0927 05:03:52.901611 (+ 1639us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:52.901628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:52.901742 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:52.902869 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:52.902877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:52.902912 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:52.903438 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.903453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.906508 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.906517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.909748 (+ 3231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.909765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.913044 (+ 3279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.913058 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.913347 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.913380 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:03:52.913484 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:03:52.913628 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":1346,"threads_started":1}
I20260927 05:03:52.914183 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.900258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.20.169.65:36061)
0927 05:03:52.900883 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:52.900891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:52.901907 (+ 1016us) server_negotiation.cc:407] Connection header received
0927 05:03:52.902074 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:52.902085 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:52.902384 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:52.902588 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:52.903714 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.906248 (+ 2534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.910062 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.912715 (+ 2653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.912923 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.913269 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:52.913570 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 05:03:52.913768 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":293,"threads_started":1}
I20260927 05:03:52.918519 6694 tablet_service.cc:1467] Processing CreateTablet for tablet 83712169acfd448ea66a266ef0df2f4e (DEFAULT_TABLE table=test-table [id=f4e074add657440591f4a1c8658f5b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:52.918577 6825 tablet_service.cc:1467] Processing CreateTablet for tablet 83712169acfd448ea66a266ef0df2f4e (DEFAULT_TABLE table=test-table [id=f4e074add657440591f4a1c8658f5b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:52.919849 6694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83712169acfd448ea66a266ef0df2f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:52.920022 6825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83712169acfd448ea66a266ef0df2f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:52.923125 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.901653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34123 (local address 127.0.0.1:55502)
0927 05:03:52.909774 (+ 8121us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:52.909794 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:52.909897 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:52.915412 (+ 5515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:52.915417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:52.915444 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:52.915614 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.915627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.917728 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.917735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.919782 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:52.919800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.922228 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.922236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:52.922443 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.922485 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:03:52.922592 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:03:52.922739 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7996,"thread_start_us":12349,"threads_started":1}
I20260927 05:03:52.923403 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:52.902010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.20.169.67:34123)
0927 05:03:52.913696 (+ 11686us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:52.913715 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:52.913750 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:03:52.913876 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:52.913886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:52.915190 (+ 1304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:52.915409 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:52.915820 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.917530 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.920050 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:52.921613 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:52.921782 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:52.922120 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:52.922663 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 05:03:52.922842 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11498,"thread_start_us":11280,"threads_started":1}
I20260927 05:03:52.927278 6955 tablet_service.cc:1467] Processing CreateTablet for tablet 83712169acfd448ea66a266ef0df2f4e (DEFAULT_TABLE table=test-table [id=f4e074add657440591f4a1c8658f5b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:52.929265 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83712169acfd448ea66a266ef0df2f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:52.931488 6759 heartbeater.cc:503] Master 127.20.169.126:36519 was elected leader, sending a full tablet report...
I20260927 05:03:52.935740 7038 tablet_bootstrap.cc:492] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Bootstrap starting.
I20260927 05:03:52.939190 7038 tablet_bootstrap.cc:654] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:52.939610 7039 tablet_bootstrap.cc:492] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: Bootstrap starting.
I20260927 05:03:52.941671 7038 log.cc:824] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:52.943266 7039 tablet_bootstrap.cc:654] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:52.944491 7038 tablet_bootstrap.cc:492] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: No bootstrap required, opened a new log
I20260927 05:03:52.944506 7039 log.cc:824] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:52.944797 7038 ts_tablet_manager.cc:1397] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 05:03:52.946648 7041 tablet_bootstrap.cc:492] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Bootstrap starting.
I20260927 05:03:52.947692 7039 tablet_bootstrap.cc:492] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: No bootstrap required, opened a new log
I20260927 05:03:52.947956 7039 ts_tablet_manager.cc:1397] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 05:03:52.950954 7041 tablet_bootstrap.cc:654] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:52.952207 7041 log.cc:824] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:52.955516 7039 raft_consensus.cc:348] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:52.955910 7039 raft_consensus.cc:374] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:52.956007 7039 raft_consensus.cc:729] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adeea7d6fc64f9db478bee5b085a6d6, State: Initialized, Role: FOLLOWER
I20260927 05:03:52.956317 7041 tablet_bootstrap.cc:492] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: No bootstrap required, opened a new log
I20260927 05:03:52.956586 7041 ts_tablet_manager.cc:1397] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 05:03:52.956619 7038 raft_consensus.cc:348] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:52.956707 7039 consensus_queue.cc:260] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:52.957118 7038 raft_consensus.cc:374] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:52.957227 7038 raft_consensus.cc:729] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f0fd4e46fd41e78b7ceb4212d5de9a, State: Initialized, Role: FOLLOWER
I20260927 05:03:52.957937 7038 consensus_queue.cc:260] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:52.958690 7039 ts_tablet_manager.cc:1428] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 05:03:52.959900 6890 heartbeater.cc:503] Master 127.20.169.126:36519 was elected leader, sending a full tablet report...
I20260927 05:03:52.960151 7038 ts_tablet_manager.cc:1428] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:03:52.966622 7041 raft_consensus.cc:348] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:52.967111 7041 raft_consensus.cc:374] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:52.967226 7041 raft_consensus.cc:729] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21adfdff5ab74399b1a96b25b7a112b1, State: Initialized, Role: FOLLOWER
I20260927 05:03:52.967815 7041 consensus_queue.cc:260] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:52.968886 7020 heartbeater.cc:503] Master 127.20.169.126:36519 was elected leader, sending a full tablet report...
I20260927 05:03:52.969491 7041 ts_tablet_manager.cc:1428] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260927 05:03:53.044318 7045 raft_consensus.cc:482] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:53.044772 7045 raft_consensus.cc:504] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:53.046711 7045 leader_election.cc:288] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6adeea7d6fc64f9db478bee5b085a6d6 (127.20.169.65:36061), 21adfdff5ab74399b1a96b25b7a112b1 (127.20.169.67:34123)
I20260927 05:03:53.054754 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.047165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.20.169.66:33183)
0927 05:03:53.047836 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.047854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:53.048044 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:53.048535 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:53.048539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:53.048566 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:53.049018 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:53.049031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.050473 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.050481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:53.052552 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:53.052566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.053964 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.053970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:53.054168 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.054207 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:03:53.054321 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 05:03:53.054412 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":1,"thread_start_us":381,"threads_started":1}
I20260927 05:03:53.054801 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.047213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:33183 (local address 127.20.169.65:36061)
0927 05:03:53.047486 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:53.047495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:53.047987 (+ 492us) server_negotiation.cc:407] Connection header received
0927 05:03:53.048239 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:53.048247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:53.048379 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:53.048487 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:53.049222 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.050299 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.052774 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.053767 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.053915 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.053970 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:53.054392 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 05:03:53.054545 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:03:53.054994 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.048201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:49483 (local address 127.20.169.67:34123)
0927 05:03:53.048434 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:53.048440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:53.048799 (+ 359us) server_negotiation.cc:407] Connection header received
0927 05:03:53.049018 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:53.049023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:53.049105 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:53.049196 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:53.049868 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.050994 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.053166 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.054023 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.054183 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.054248 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:53.054584 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 05:03:53.054712 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 05:03:53.055105 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.047997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:34123 (local address 127.20.169.66:49483)
0927 05:03:53.048682 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.048700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:53.048863 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:53.049232 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:53.049236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:53.049263 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:53.049654 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:53.049666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.051155 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.051159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:53.052952 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:53.052969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.054205 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.054209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:53.054410 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.054444 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:03:53.054538 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 05:03:53.054663 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":50,"thread_start_us":322,"threads_started":1}
I20260927 05:03:53.056118 6975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83712169acfd448ea66a266ef0df2f4e" candidate_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21adfdff5ab74399b1a96b25b7a112b1" is_pre_election: true
I20260927 05:03:53.056115 6714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83712169acfd448ea66a266ef0df2f4e" candidate_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" is_pre_election: true
I20260927 05:03:53.056679 6714 raft_consensus.cc:2454] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f0fd4e46fd41e78b7ceb4212d5de9a in term 0.
I20260927 05:03:53.056679 6975 raft_consensus.cc:2454] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f0fd4e46fd41e78b7ceb4212d5de9a in term 0.
I20260927 05:03:53.057412 6778 leader_election.cc:302] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f0fd4e46fd41e78b7ceb4212d5de9a, 6adeea7d6fc64f9db478bee5b085a6d6; no voters:
I20260927 05:03:53.057776 7045 raft_consensus.cc:2781] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:03:53.057961 7045 raft_consensus.cc:482] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:03:53.058109 7045 raft_consensus.cc:3037] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:53.060736 7045 raft_consensus.cc:504] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:53.061484 7045 leader_election.cc:288] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [CANDIDATE]: Term 1 election: Requested vote from peers 6adeea7d6fc64f9db478bee5b085a6d6 (127.20.169.65:36061), 21adfdff5ab74399b1a96b25b7a112b1 (127.20.169.67:34123)
I20260927 05:03:53.061962 6714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83712169acfd448ea66a266ef0df2f4e" candidate_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adeea7d6fc64f9db478bee5b085a6d6"
I20260927 05:03:53.062026 6975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83712169acfd448ea66a266ef0df2f4e" candidate_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21adfdff5ab74399b1a96b25b7a112b1"
I20260927 05:03:53.062150 6714 raft_consensus.cc:3037] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:53.062311 6975 raft_consensus.cc:3037] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:53.064474 6714 raft_consensus.cc:2454] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f0fd4e46fd41e78b7ceb4212d5de9a in term 1.
I20260927 05:03:53.064982 6778 leader_election.cc:302] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f0fd4e46fd41e78b7ceb4212d5de9a, 6adeea7d6fc64f9db478bee5b085a6d6; no voters:
I20260927 05:03:53.064996 6975 raft_consensus.cc:2454] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f0fd4e46fd41e78b7ceb4212d5de9a in term 1.
I20260927 05:03:53.065296 7045 raft_consensus.cc:2781] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:53.065831 7045 raft_consensus.cc:686] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 1 LEADER]: Becoming Leader. State: Replica: 61f0fd4e46fd41e78b7ceb4212d5de9a, State: Running, Role: LEADER
I20260927 05:03:53.066390 7045 consensus_queue.cc:237] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } }
I20260927 05:03:53.071869 6573 catalog_manager.cc:5495] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 61f0fd4e46fd41e78b7ceb4212d5de9a (127.20.169.66). New cstate: current_term: 1 leader_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f0fd4e46fd41e78b7ceb4212d5de9a" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 } health_report { overall_health: UNKNOWN } } }
W20260927 05:03:53.093612 7021 tablet.cc:2369] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:03:53.121678 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.116075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.20.169.66:38593)
0927 05:03:53.116284 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:53.116291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:53.116387 (+ 96us) server_negotiation.cc:407] Connection header received
0927 05:03:53.116631 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:53.116637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:53.116760 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:53.116960 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:03:53.117330 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.118125 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.119520 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.119919 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.120050 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.120568 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:03:53.121112 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:03:53.121275 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:53.121333 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 05:03:53.121411 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 05:03:53.124130 6891 tablet.cc:2369] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:03:53.130167 6714 raft_consensus.cc:1261] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 1 FOLLOWER]: Refusing update from remote peer 61f0fd4e46fd41e78b7ceb4212d5de9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:53.130167 6975 raft_consensus.cc:1261] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 1 FOLLOWER]: Refusing update from remote peer 61f0fd4e46fd41e78b7ceb4212d5de9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:53.131103 7049 consensus_queue.cc:1035] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21adfdff5ab74399b1a96b25b7a112b1" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:53.131457 7045 consensus_queue.cc:1035] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adeea7d6fc64f9db478bee5b085a6d6" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 36061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:53.143625 7053 mvcc.cc:205] Tried to move back new op lower bound from 7333828334091096064 to 7333828333845278720. Current Snapshot: MvccSnapshot[applied={T|T < 7333828334091096064}]
I20260927 05:03:53.145526 7052 mvcc.cc:205] Tried to move back new op lower bound from 7333828334091096064 to 7333828333845278720. Current Snapshot: MvccSnapshot[applied={T|T < 7333828334091096064 or (T in {7333828334091096064})}]
I20260927 05:03:53.154492 6572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:53.154768 6572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:53.158666 6572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d17ec8813184341ace6fce679c30c85: Sending DeleteTablet for 3 replicas of tablet 83712169acfd448ea66a266ef0df2f4e
I20260927 05:03:53.159935 6825 tablet_service.cc:1514] Processing DeleteTablet for tablet 83712169acfd448ea66a266ef0df2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:53 UTC) from {username='slave'} at 127.0.0.1:58678
I20260927 05:03:53.159902 6955 tablet_service.cc:1514] Processing DeleteTablet for tablet 83712169acfd448ea66a266ef0df2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:53 UTC) from {username='slave'} at 127.0.0.1:55502
I20260927 05:03:53.160631 6694 tablet_service.cc:1514] Processing DeleteTablet for tablet 83712169acfd448ea66a266ef0df2f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:53 UTC) from {username='slave'} at 127.0.0.1:56492
I20260927 05:03:53.161276 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6632
I20260927 05:03:53.179862 7064 tablet_replica.cc:330] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: stopping tablet replica
I20260927 05:03:53.180362 7064 raft_consensus.cc:2229] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:53.180981 7064 raft_consensus.cc:2258] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:53.183485 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.162538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.20.169.65:36061)
0927 05:03:53.162710 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:53.162716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:53.172910 (+ 10194us) server_negotiation.cc:407] Connection header received
0927 05:03:53.173400 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:53.173406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:53.173531 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:53.173622 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:53.174340 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.175559 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.177652 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.178357 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.178455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.181690 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:53.181702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:53.181746 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:53.182542 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:53.182603 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:53.182628 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:53.182712 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:53.182983 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 05:03:53.183100 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:03:53.184379 7064 ts_tablet_manager.cc:1905] T 83712169acfd448ea66a266ef0df2f4e P 6adeea7d6fc64f9db478bee5b085a6d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:53.184684 6754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:53.185364 7062 tablet_replica.cc:330] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: stopping tablet replica
I20260927 05:03:53.185792 7062 raft_consensus.cc:2229] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:53.186163 7063 tablet_replica.cc:330] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: stopping tablet replica
I20260927 05:03:53.186481 7062 raft_consensus.cc:2258] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:53.186802 7063 raft_consensus.cc:2229] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:03:53.187661 7063 raft_consensus.cc:2258] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:53.188711 7062 ts_tablet_manager.cc:1905] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:53.190289 7063 ts_tablet_manager.cc:1905] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:53.196242 7062 ts_tablet_manager.cc:1918] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:53.196405 7062 log.cc:1196] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/83712169acfd448ea66a266ef0df2f4e
I20260927 05:03:53.196997 7062 ts_tablet_manager.cc:1939] T 83712169acfd448ea66a266ef0df2f4e P 21adfdff5ab74399b1a96b25b7a112b1: Deleting consensus metadata
I20260927 05:03:53.198469 7063 ts_tablet_manager.cc:1918] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:53.198696 7063 log.cc:1196] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/83712169acfd448ea66a266ef0df2f4e
I20260927 05:03:53.198894 6557 catalog_manager.cc:4841] TS 21adfdff5ab74399b1a96b25b7a112b1 (127.20.169.67:34123): tablet 83712169acfd448ea66a266ef0df2f4e (table test-table [id=f4e074add657440591f4a1c8658f5b0a]) successfully deleted
I20260927 05:03:53.199412 7063 ts_tablet_manager.cc:1939] T 83712169acfd448ea66a266ef0df2f4e P 61f0fd4e46fd41e78b7ceb4212d5de9a: Deleting consensus metadata
I20260927 05:03:53.201215 6557 catalog_manager.cc:4841] TS 61f0fd4e46fd41e78b7ceb4212d5de9a (127.20.169.66:38593): tablet 83712169acfd448ea66a266ef0df2f4e (table test-table [id=f4e074add657440591f4a1c8658f5b0a]) successfully deleted
I20260927 05:03:53.281953 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6632
W20260927 05:03:53.307966 6557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:36061 (error 108)
I20260927 05:03:53.308704 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6763
I20260927 05:03:53.310130 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.309495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56506)
0927 05:03:53.309781 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.309909 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260927 05:03:53.310401 6557 catalog_manager.cc:4584] TS 6adeea7d6fc64f9db478bee5b085a6d6 (127.20.169.65:36061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83712169acfd448ea66a266ef0df2f4e: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
I20260927 05:03:53.317131 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.309566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.20.169.66:38593)
0927 05:03:53.310179 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:53.310187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:53.310207 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:03:53.310273 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:53.310281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:53.310402 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:53.310525 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:53.311027 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.311912 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.314031 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.314457 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.314553 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.315672 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:53.315684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:53.315697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:53.316307 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:53.316385 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:53.316405 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:53.316489 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:53.316627 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 05:03:53.316744 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477}
I20260927 05:03:53.317782 6885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:53.364095 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.363587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56508)
0927 05:03:53.363810 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.363932 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 05:03:53.365378 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.364944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56522)
0927 05:03:53.365137 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.365222 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:03:53.426121 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.425641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56528)
0927 05:03:53.425872 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.425970 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 05:03:53.427287 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.426946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56536)
0927 05:03:53.427095 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.427170 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 05:03:53.444741 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6763
I20260927 05:03:53.465180 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6893
I20260927 05:03:53.472329 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.466048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.20.169.67:34123)
0927 05:03:53.466227 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:53.466231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:53.466249 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:03:53.466470 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:53.466475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:53.466583 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:53.466676 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:53.467115 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.467905 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.469275 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:53.469675 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:53.469840 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:53.470879 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:53.470888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:53.470904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:53.471474 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:53.471523 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:53.471549 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:53.471641 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:53.471885 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 05:03:53.471983 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:03:53.473066 7015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:53.508441 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.507894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56552)
0927 05:03:53.508136 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.508253 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:03:53.509879 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.509355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56556)
0927 05:03:53.509589 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.509690 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 05:03:53.562449 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6893
I20260927 05:03:53.582032 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 6542
I20260927 05:03:53.583278 6603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:53.693030 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 6542
I20260927 05:03:53.693080 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:53.692317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:36061 (local address 127.0.0.1:56570)
0927 05:03:53.692537 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:53.692801 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:36061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
2026-09-27T05:03:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2852 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:03:53 dist-test-slave-39f2 krb5kdc[7073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:03:53 dist-test-slave-39f2 krb5kdc[7073](info): set up 2 sockets
Sep 27 05:03:53 dist-test-slave-39f2 krb5kdc[7073](info): commencing operation
krb5kdc: starting...
W20260927 05:03:54.913496 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:03:55 dist-test-slave-39f2 krb5kdc[7073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:03:55.169703 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:37979
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:55.514144 7086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:55.514439 7086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:55.514528 7086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:55.521587 7086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:03:55.521740 7086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:55.521842 7086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:55.521938 7086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:03:55.522004 7086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:03:55.531634 7086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:37979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:55.532912 7086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:55.534192 7086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:55.545809 7091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:55.545883 7092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:55.546105 7094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:55.546841 7086 server_base.cc:1034] running on GCE node
I20260927 05:03:55.547694 7086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:55.549252 7086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:55.550500 7086 hybrid_clock.cc:648] HybridClock initialized: now 1790485435550430 us; error 33 us; skew 500 ppm
I20260927 05:03:55.550881 7086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:55 dist-test-slave-39f2 krb5kdc[7073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485435, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:55.554526 7086 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:03:55.556217 7086 webserver.cc:460] Webserver started at http://127.20.169.126:33571/ using document root <none> and password file <none>
I20260927 05:03:55.557050 7086 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:55.557224 7086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:55.557571 7086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:55.560976 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "3c5711e4b3674012b0bce65cefb10cf6"
format_stamp: "Formatted at 2026-09-27 05:03:55 on dist-test-slave-39f2"
server_key: "c8e5e35a5b8d269178389057434692cd"
server_key_iv: "846bc729e91573f1ebd68c417210906f"
server_key_version: "encryptionkey@0"
I20260927 05:03:55.561807 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "3c5711e4b3674012b0bce65cefb10cf6"
format_stamp: "Formatted at 2026-09-27 05:03:55 on dist-test-slave-39f2"
server_key: "c8e5e35a5b8d269178389057434692cd"
server_key_iv: "846bc729e91573f1ebd68c417210906f"
server_key_version: "encryptionkey@0"
I20260927 05:03:55.567584 7086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 05:03:55.572448 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:55.573905 7086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:55.574098 7086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "3c5711e4b3674012b0bce65cefb10cf6"
format_stamp: "Formatted at 2026-09-27 05:03:55 on dist-test-slave-39f2"
server_key: "c8e5e35a5b8d269178389057434692cd"
server_key_iv: "846bc729e91573f1ebd68c417210906f"
server_key_version: "encryptionkey@0"
I20260927 05:03:55.574290 7086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:55.638767 7086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:55.643165 7086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:55.643487 7086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:55.663349 7086 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:37979
I20260927 05:03:55.663349 7152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:37979 every 8 connection(s)
I20260927 05:03:55.664690 7086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:03:55.666188 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7086
I20260927 05:03:55.666380 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:03:55.666978 21157 external_mini_cluster.cc:1428] Setting key e2cfc97071a70cbb5212ba7d696cb8e7
I20260927 05:03:55.669484 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:03:55 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:55.682839 7153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6: Bootstrap starting.
I20260927 05:03:55.687099 7153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:55.688254 7153 log.cc:824] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:55.691830 7153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6: No bootstrap required, opened a new log
I20260927 05:03:55.692216 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:55.668904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.20.169.126:37979)
0927 05:03:55.669676 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:55.669688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:55.669729 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:03:55.671372 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:55.671393 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:55.671766 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:55.672108 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:55.673973 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:55.675293 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:55.677216 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:55.678000 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:55.678124 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:55.683385 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:55.683394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:55.683412 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:55.683453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:55.687637 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:55.688421 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:55.688429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:55.688436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:55.688630 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:55.689160 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:55.689163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:55.689165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:55.689676 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:55.690025 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:55.690434 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 05:03:55.690868 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":368,"threads_started":1}
I20260927 05:03:55.700421 7153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5711e4b3674012b0bce65cefb10cf6" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37979 } }
I20260927 05:03:55.700757 7153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:55.700905 7153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5711e4b3674012b0bce65cefb10cf6, State: Initialized, Role: FOLLOWER
I20260927 05:03:55.701460 7153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5711e4b3674012b0bce65cefb10cf6" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37979 } }
I20260927 05:03:55.701812 7153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:03:55.701921 7153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:03:55.702049 7153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:55.704082 7153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5711e4b3674012b0bce65cefb10cf6" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37979 } }
I20260927 05:03:55.704653 7153 leader_election.cc:302] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c5711e4b3674012b0bce65cefb10cf6; no voters:
I20260927 05:03:55.705328 7153 leader_election.cc:288] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:03:55.705499 7158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:55.706264 7158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5711e4b3674012b0bce65cefb10cf6, State: Running, Role: LEADER
I20260927 05:03:55.706808 7158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5711e4b3674012b0bce65cefb10cf6" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37979 } }
I20260927 05:03:55.707432 7153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:03:55.710103 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c5711e4b3674012b0bce65cefb10cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5711e4b3674012b0bce65cefb10cf6" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37979 } } }
I20260927 05:03:55.710444 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:55.711283 7160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c5711e4b3674012b0bce65cefb10cf6. Latest consensus state: current_term: 1 leader_uuid: "3c5711e4b3674012b0bce65cefb10cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5711e4b3674012b0bce65cefb10cf6" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 37979 } } }
I20260927 05:03:55.711690 7160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6 [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:55.714025 7164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:03:55.719660 7164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:03:55.728261 7164 catalog_manager.cc:1347] Generated new cluster ID: 4b1bc8f9da514310ba30496219397b86
I20260927 05:03:55.728381 7164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:03:55.747462 7164 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:03:55.748785 7164 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:03:55.755640 7164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6: Generated new TSK 0
I20260927 05:03:55.756436 7164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:03:55.826099 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:37979
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:56.204241 7179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:56.204505 7179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:56.204602 7179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:56.212193 7179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:56.212292 7179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:56.212404 7179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:03:56.222174 7179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:56.223371 7179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:56.224601 7179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:56.237669 7187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:56.237756 7185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:56.237756 7184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:56.238454 7179 server_base.cc:1034] running on GCE node
I20260927 05:03:56.239061 7179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:56.240023 7179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:56.241246 7179 hybrid_clock.cc:648] HybridClock initialized: now 1790485436241181 us; error 65 us; skew 500 ppm
I20260927 05:03:56.241556 7179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:56 dist-test-slave-39f2 krb5kdc[7073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:56.244690 7179 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:03:56.246251 7179 webserver.cc:460] Webserver started at http://127.20.169.65:39763/ using document root <none> and password file <none>
I20260927 05:03:56.246938 7179 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:56.247084 7179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:56.247403 7179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:56.250340 7179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "c97570a35a4a45f983a8f39695fe1290"
format_stamp: "Formatted at 2026-09-27 05:03:56 on dist-test-slave-39f2"
server_key: "c72521a042e0909071f6fd2d910ab3a1"
server_key_iv: "b7ce79c5b937ae35f7a857a61c5d2486"
server_key_version: "encryptionkey@0"
I20260927 05:03:56.251104 7179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "c97570a35a4a45f983a8f39695fe1290"
format_stamp: "Formatted at 2026-09-27 05:03:56 on dist-test-slave-39f2"
server_key: "c72521a042e0909071f6fd2d910ab3a1"
server_key_iv: "b7ce79c5b937ae35f7a857a61c5d2486"
server_key_version: "encryptionkey@0"
I20260927 05:03:56.256313 7179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 05:03:56.259964 7194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:56.261409 7179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:56.261577 7179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "c97570a35a4a45f983a8f39695fe1290"
format_stamp: "Formatted at 2026-09-27 05:03:56 on dist-test-slave-39f2"
server_key: "c72521a042e0909071f6fd2d910ab3a1"
server_key_iv: "b7ce79c5b937ae35f7a857a61c5d2486"
server_key_version: "encryptionkey@0"
I20260927 05:03:56.261781 7179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:56.282235 7179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:56.286589 7179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:56.286892 7179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:56.287835 7179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:56.289284 7179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:56.289388 7179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:56.289556 7179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:56.289629 7179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:56.323715 7179 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:42427
I20260927 05:03:56.323760 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:42427 every 8 connection(s)
I20260927 05:03:56.325063 7179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:03:56.327329 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7179
I20260927 05:03:56.327533 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:03:56.328091 21157 external_mini_cluster.cc:1428] Setting key ed0f0b8a68cababa5bdcd707bb20998b
Sep 27 05:03:56 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:56.349418 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:56.327816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:44535 (local address 127.20.169.126:37979)
0927 05:03:56.328356 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:56.328361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:56.329454 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 05:03:56.331559 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:56.331564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:56.331652 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:56.331777 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:56.333317 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.334051 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.335608 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.336337 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.336489 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:56.343667 (+ 7178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:56.343672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:56.343675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:56.343705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:56.346269 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:56.347038 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:56.347043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:56.347047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:56.347188 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:56.347634 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:56.347639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:56.347641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:56.347904 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:56.348062 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:56.348752 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 05:03:56.349021 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":274,"threads_started":1}
I20260927 05:03:56.350431 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:56.328369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:37979 (local address 127.20.169.65:44535)
0927 05:03:56.329259 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:56.329334 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:56.331300 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:56.332007 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:56.332015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:56.332523 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:56.333149 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:56.333166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.334197 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.334207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:56.335429 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:56.335442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.336562 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.336570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:56.336902 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:56.340754 (+ 3852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:56.340794 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:56.343439 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:56.346455 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:56.346464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:56.346482 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:56.346906 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:56.347304 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:56.347310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:56.347314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:56.347508 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:56.348034 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:56.348042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:56.348390 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 05:03:56.348655 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 05:03:56.349263 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":398,"threads_started":1}
I20260927 05:03:56.352172 7307 heartbeater.cc:347] Connected to a master server at 127.20.169.126:37979
I20260927 05:03:56.352483 7307 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:56.353247 7307 heartbeater.cc:511] Master 127.20.169.126:37979 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 05:03:56.355453 7118 ts_manager.cc:194] Registered new tserver with Master: c97570a35a4a45f983a8f39695fe1290 (127.20.169.65:42427)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
I20260927 05:03:56.357077 7118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:44535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:03:56.387770 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:37979
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:56.726802 7313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:56.727074 7313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:56.727169 7313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:56.734196 7313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:56.734297 7313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:56.734407 7313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:03:56.743729 7313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:56.744923 7313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:56.746177 7313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:56.758985 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:56.759042 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:56.759057 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:56.760190 7313 server_base.cc:1034] running on GCE node
I20260927 05:03:56.760931 7313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:56.761848 7313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:56.763077 7313 hybrid_clock.cc:648] HybridClock initialized: now 1790485436763019 us; error 74 us; skew 500 ppm
I20260927 05:03:56.763391 7313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:56 dist-test-slave-39f2 krb5kdc[7073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:56.767016 7313 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:03:56.768520 7313 webserver.cc:460] Webserver started at http://127.20.169.66:38345/ using document root <none> and password file <none>
I20260927 05:03:56.769273 7313 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:56.769418 7313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:56.769726 7313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:56.772581 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "828daa3e05a24505a0e752f4afed45b5"
format_stamp: "Formatted at 2026-09-27 05:03:56 on dist-test-slave-39f2"
server_key: "62711cf7fff2c66c3bd6b77ee2d63a6f"
server_key_iv: "eefef50b51bfa414413d9274a47376c4"
server_key_version: "encryptionkey@0"
I20260927 05:03:56.773453 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "828daa3e05a24505a0e752f4afed45b5"
format_stamp: "Formatted at 2026-09-27 05:03:56 on dist-test-slave-39f2"
server_key: "62711cf7fff2c66c3bd6b77ee2d63a6f"
server_key_iv: "eefef50b51bfa414413d9274a47376c4"
server_key_version: "encryptionkey@0"
I20260927 05:03:56.778492 7313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:03:56.782079 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:56.783686 7313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:03:56.783918 7313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "828daa3e05a24505a0e752f4afed45b5"
format_stamp: "Formatted at 2026-09-27 05:03:56 on dist-test-slave-39f2"
server_key: "62711cf7fff2c66c3bd6b77ee2d63a6f"
server_key_iv: "eefef50b51bfa414413d9274a47376c4"
server_key_version: "encryptionkey@0"
I20260927 05:03:56.784174 7313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:56.806313 7313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:56.810789 7313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:56.811088 7313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:56.812088 7313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:56.813504 7313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:56.813606 7313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:56.813756 7313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:56.813828 7313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:56.848160 7313 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:42575
I20260927 05:03:56.848206 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:42575 every 8 connection(s)
I20260927 05:03:56.849550 7313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:03:56.854933 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7313
I20260927 05:03:56.855121 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:03:56.855548 21157 external_mini_cluster.cc:1428] Setting key 485b36ddd5d8ec4611fc9d54c8fc1045
Sep 27 05:03:56 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485436, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:56.871299 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:56.852216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:56663 (local address 127.20.169.126:37979)
0927 05:03:56.852671 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:56.852676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:56.853691 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 05:03:56.855608 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:56.855613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:56.855708 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:56.855826 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:56.857621 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.858698 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.860238 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.861017 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.861159 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:56.864686 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:56.864719 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:56.864725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:56.864772 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:56.867886 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:56.868874 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:56.868880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:56.868885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:56.869054 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:56.869531 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:56.869534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:56.869536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:56.869840 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:56.869947 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:56.870850 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 05:03:56.871026 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260927 05:03:56.872797 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:56.852782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:37979 (local address 127.20.169.66:56663)
0927 05:03:56.853522 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:56.853565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:56.855354 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:56.856113 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:56.856125 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:56.856693 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:56.857422 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:56.857449 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.858863 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.858867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:56.860091 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:56.860103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:56.861232 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:56.861236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:56.861434 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:56.862108 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:56.862133 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:56.864459 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:56.868022 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:56.868034 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:56.868051 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:56.868676 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:56.869166 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:56.869174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:56.869181 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:56.869419 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:56.869976 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:56.869998 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:56.870433 (+ 435us) client_negotiation.cc:769] Sending connection context
0927 05:03:56.870739 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 05:03:56.871335 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":268,"threads_started":1}
I20260927 05:03:56.874336 7441 heartbeater.cc:347] Connected to a master server at 127.20.169.126:37979
I20260927 05:03:56.874645 7441 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:56.875379 7441 heartbeater.cc:511] Master 127.20.169.126:37979 requested a full tablet report, sending...
I20260927 05:03:56.877183 7118 ts_manager.cc:194] Registered new tserver with Master: 828daa3e05a24505a0e752f4afed45b5 (127.20.169.66:42575)
I20260927 05:03:56.878436 7118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:56663
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:03:56.915755 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:37979
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:03:57.243883 7447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:57.244223 7447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:57.244398 7447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:57.251950 7447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:57.252111 7447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:57.252300 7447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:03:57.262202 7447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:57.263579 7447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:57.264998 7447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:57.278923 7453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:57.279014 7455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:57.278959 7452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:57.280267 7447 server_base.cc:1034] running on GCE node
I20260927 05:03:57.281139 7447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:57.282240 7447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:57.283512 7447 hybrid_clock.cc:648] HybridClock initialized: now 1790485437283453 us; error 63 us; skew 500 ppm
I20260927 05:03:57.283902 7447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:03:57 dist-test-slave-39f2 krb5kdc[7073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:03:57.287604 7447 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:03:57.289271 7447 webserver.cc:460] Webserver started at http://127.20.169.67:42033/ using document root <none> and password file <none>
I20260927 05:03:57.290063 7447 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:57.290206 7447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:57.290520 7447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:57.293474 7447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "74238dfbd935469b99c860bb4d809f19"
format_stamp: "Formatted at 2026-09-27 05:03:57 on dist-test-slave-39f2"
server_key: "48e0eb9aa0fe1d8984c3baf95a52b5ee"
server_key_iv: "1191d80a7fe6ed8a00ad3734149500e5"
server_key_version: "encryptionkey@0"
I20260927 05:03:57.294245 7447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "74238dfbd935469b99c860bb4d809f19"
format_stamp: "Formatted at 2026-09-27 05:03:57 on dist-test-slave-39f2"
server_key: "48e0eb9aa0fe1d8984c3baf95a52b5ee"
server_key_iv: "1191d80a7fe6ed8a00ad3734149500e5"
server_key_version: "encryptionkey@0"
I20260927 05:03:57.299712 7447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:03:57.303503 7462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:57.305262 7447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:03:57.305445 7447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "74238dfbd935469b99c860bb4d809f19"
format_stamp: "Formatted at 2026-09-27 05:03:57 on dist-test-slave-39f2"
server_key: "48e0eb9aa0fe1d8984c3baf95a52b5ee"
server_key_iv: "1191d80a7fe6ed8a00ad3734149500e5"
server_key_version: "encryptionkey@0"
I20260927 05:03:57.305673 7447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:57.335605 7447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:57.339498 7447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:57.339781 7447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:57.340786 7447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:57.342123 7447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:57.342226 7447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:57.342368 7447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:57.342440 7447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:57.360072 7307 heartbeater.cc:503] Master 127.20.169.126:37979 was elected leader, sending a full tablet report...
I20260927 05:03:57.377447 7447 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:41399
I20260927 05:03:57.377490 7574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:41399 every 8 connection(s)
I20260927 05:03:57.378991 7447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:03:57.381412 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7447
I20260927 05:03:57.381634 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:03:57.382272 21157 external_mini_cluster.cc:1428] Setting key 62cac1b08ad437a3aee990d370789fc4
Sep 27 05:03:57 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485437, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:03:57.401070 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.381667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:47891 (local address 127.20.169.126:37979)
0927 05:03:57.382220 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.382227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.383379 (+ 1152us) server_negotiation.cc:407] Connection header received
0927 05:03:57.385993 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.385997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.386074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.386163 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:57.387808 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.388683 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.390327 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.390740 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.390852 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.394300 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:57.394311 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:57.394315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:57.394350 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:57.397792 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.398739 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.398743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.398745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.398899 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.399410 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.399414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.399416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.399693 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:57.399827 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.400544 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.400707 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":238,"threads_started":1}
I20260927 05:03:57.401949 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.382278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:37979 (local address 127.20.169.67:47891)
0927 05:03:57.383168 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:57.383231 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:57.385733 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:57.386377 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:57.386386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:57.386964 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:57.387618 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.387639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.388899 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.388905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.390166 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.390178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.390935 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.390940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.391126 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.391848 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:03:57.391869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:57.394059 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:57.397942 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:57.397960 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:57.397995 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:57.398607 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:57.399027 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:03:57.399031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:03:57.399033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:03:57.399227 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:03:57.399819 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:57.399825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:57.400156 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 05:03:57.400428 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 05:03:57.400802 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":317,"threads_started":1}
I20260927 05:03:57.403378 7575 heartbeater.cc:347] Connected to a master server at 127.20.169.126:37979
I20260927 05:03:57.403781 7575 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:57.404454 7575 heartbeater.cc:511] Master 127.20.169.126:37979 requested a full tablet report, sending...
I20260927 05:03:57.405925 7118 ts_manager.cc:194] Registered new tserver with Master: 74238dfbd935469b99c860bb4d809f19 (127.20.169.67:41399)
I20260927 05:03:57.406729 7118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:47891
I20260927 05:03:57.410773 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:03:57.429805 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.415396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.20.169.126:37979)
0927 05:03:57.415578 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.415583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.416096 (+ 513us) server_negotiation.cc:407] Connection header received
0927 05:03:57.416354 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.416360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.416463 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.416589 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:57.418499 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.419319 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.420974 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.421431 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.421602 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.424506 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:57.424513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:57.424518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:57.424559 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:57.426962 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.427722 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.427734 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.427737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.427859 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.428306 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.428313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.428320 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.428606 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:57.428943 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.429306 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.429494 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:03:57.433736 7118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:03:57.440691 7118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:03:57.462646 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.452506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42427 (local address 127.0.0.1:35010)
0927 05:03:57.453386 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:57.453442 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:57.453625 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:57.455539 (+ 1914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:57.455547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:57.455581 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:57.456188 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.456204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.458158 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.458164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.460306 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.460318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.461626 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.461632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.461835 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.461904 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 05:03:57.462050 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 05:03:57.462231 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":70,"spinlock_wait_cycles":16512,"thread_start_us":1172,"threads_started":1}
I20260927 05:03:57.463421 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.452757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.20.169.65:42427)
0927 05:03:57.454864 (+ 2107us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.454876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.454906 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:03:57.455006 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.455013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.455279 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.456368 (+ 1089us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:57.456627 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.458035 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.460455 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.461434 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.461589 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.461832 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.462087 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.462248 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1937,"thread_start_us":1777,"threads_started":1}
I20260927 05:03:57.464773 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.452535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:41399 (local address 127.0.0.1:37854)
0927 05:03:57.455610 (+ 3075us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:57.455629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:57.455768 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:57.456674 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:57.456680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:57.456705 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:57.457209 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.457220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.459235 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.459242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.462080 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.462090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.463935 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.463942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.464109 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.464144 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:03:57.464230 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:03:57.464356 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2960,"mutex_wait_us":159,"spinlock_wait_cycles":8320,"thread_start_us":3189,"threads_started":1}
I20260927 05:03:57.464972 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.453333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.20.169.67:41399)
0927 05:03:57.454359 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.454372 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.455861 (+ 1489us) server_negotiation.cc:407] Connection header received
0927 05:03:57.455978 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.455986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.456211 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.456411 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:57.457406 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.459047 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.462257 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.463685 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.463822 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.464033 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.464322 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.464541 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":688,"threads_started":1}
I20260927 05:03:57.465411 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.453353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.20.169.66:42575)
0927 05:03:57.455154 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.455162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.456336 (+ 1174us) server_negotiation.cc:407] Connection header received
0927 05:03:57.456439 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.456447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.456651 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.456878 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:57.457606 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.459116 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.462259 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.463685 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.463812 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.463999 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.464890 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.465067 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":1543,"threads_started":1}
I20260927 05:03:57.465852 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.453263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:42575 (local address 127.0.0.1:51118)
0927 05:03:57.456060 (+ 2797us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:57.456077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:57.456208 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:57.456954 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:57.456958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:57.456987 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:57.457439 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.457452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.459266 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.459273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.462069 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.462085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.464143 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.464147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.464350 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.464380 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:03:57.464465 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 05:03:57.464585 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2665,"mutex_wait_us":195,"thread_start_us":1997,"threads_started":1}
I20260927 05:03:57.467598 7242 tablet_service.cc:1467] Processing CreateTablet for tablet c3755a0b902740a4b55ca55e72fef346 (DEFAULT_TABLE table=test-table [id=115712c51eac4e54ac9bdc885f3753c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:57.469165 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3755a0b902740a4b55ca55e72fef346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:57.469168 7510 tablet_service.cc:1467] Processing CreateTablet for tablet c3755a0b902740a4b55ca55e72fef346 (DEFAULT_TABLE table=test-table [id=115712c51eac4e54ac9bdc885f3753c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:57.469230 7376 tablet_service.cc:1467] Processing CreateTablet for tablet c3755a0b902740a4b55ca55e72fef346 (DEFAULT_TABLE table=test-table [id=115712c51eac4e54ac9bdc885f3753c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:03:57.470418 7510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3755a0b902740a4b55ca55e72fef346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:57.470569 7376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3755a0b902740a4b55ca55e72fef346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:57.482484 7594 tablet_bootstrap.cc:492] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Bootstrap starting.
I20260927 05:03:57.484676 7595 tablet_bootstrap.cc:492] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Bootstrap starting.
I20260927 05:03:57.487098 7594 tablet_bootstrap.cc:654] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:57.488482 7595 tablet_bootstrap.cc:654] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:57.488634 7594 log.cc:824] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:57.490047 7595 log.cc:824] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:57.492311 7594 tablet_bootstrap.cc:492] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: No bootstrap required, opened a new log
I20260927 05:03:57.492550 7596 tablet_bootstrap.cc:492] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Bootstrap starting.
I20260927 05:03:57.492653 7594 ts_tablet_manager.cc:1397] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 05:03:57.495128 7595 tablet_bootstrap.cc:492] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: No bootstrap required, opened a new log
I20260927 05:03:57.495468 7595 ts_tablet_manager.cc:1397] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 05:03:57.496122 7596 tablet_bootstrap.cc:654] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:57.497258 7596 log.cc:824] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:57.499636 7596 tablet_bootstrap.cc:492] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: No bootstrap required, opened a new log
I20260927 05:03:57.499922 7596 ts_tablet_manager.cc:1397] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 05:03:57.503408 7594 raft_consensus.cc:348] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.503861 7594 raft_consensus.cc:374] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:57.504005 7594 raft_consensus.cc:729] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74238dfbd935469b99c860bb4d809f19, State: Initialized, Role: FOLLOWER
I20260927 05:03:57.504664 7594 consensus_queue.cc:260] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.505892 7575 heartbeater.cc:503] Master 127.20.169.126:37979 was elected leader, sending a full tablet report...
I20260927 05:03:57.506595 7594 ts_tablet_manager.cc:1428] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 05:03:57.506357 7595 raft_consensus.cc:348] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.506789 7595 raft_consensus.cc:374] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:57.506884 7595 raft_consensus.cc:729] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97570a35a4a45f983a8f39695fe1290, State: Initialized, Role: FOLLOWER
I20260927 05:03:57.507519 7595 consensus_queue.cc:260] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.508033 7596 raft_consensus.cc:348] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.508467 7596 raft_consensus.cc:374] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:57.508605 7596 raft_consensus.cc:729] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828daa3e05a24505a0e752f4afed45b5, State: Initialized, Role: FOLLOWER
I20260927 05:03:57.509168 7596 consensus_queue.cc:260] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.510244 7441 heartbeater.cc:503] Master 127.20.169.126:37979 was elected leader, sending a full tablet report...
I20260927 05:03:57.511164 7596 ts_tablet_manager.cc:1428] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 05:03:57.513059 7595 ts_tablet_manager.cc:1428] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260927 05:03:57.524039 7600 raft_consensus.cc:482] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:03:57.524323 7600 raft_consensus.cc:504] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.525887 7600 leader_election.cc:288] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c97570a35a4a45f983a8f39695fe1290 (127.20.169.65:42427), 828daa3e05a24505a0e752f4afed45b5 (127.20.169.66:42575)
I20260927 05:03:57.532095 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.526150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:42575 (local address 127.20.169.67:34559)
0927 05:03:57.526491 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:57.526513 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:57.526691 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:57.527109 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:57.527114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:57.527144 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:57.527428 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.527440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.528578 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.528584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.530027 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.530039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.531359 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.531367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.531555 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.531590 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:03:57.531694 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:03:57.531836 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1}
I20260927 05:03:57.532145 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.526266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:34559 (local address 127.20.169.66:42575)
0927 05:03:57.526455 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.526463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.526605 (+ 142us) server_negotiation.cc:407] Connection header received
0927 05:03:57.526865 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.526870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.526957 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.527066 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:57.527573 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.528443 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.530184 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.531170 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.531284 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.531336 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.531720 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.531871 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:03:57.532563 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.526258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:39199 (local address 127.20.169.65:42427)
0927 05:03:57.526443 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.526450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.527110 (+ 660us) server_negotiation.cc:407] Connection header received
0927 05:03:57.527263 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.527267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.527343 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.527442 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:03:57.528018 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.528878 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.530724 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.531687 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.531838 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.531902 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.532250 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.532394 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:03:57.533136 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.526258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42427 (local address 127.20.169.67:39199)
0927 05:03:57.526999 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:57.527018 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:57.527140 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:57.527479 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:57.527482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:57.527522 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:03:57.527864 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.527877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.529013 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.529016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.530576 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:57.530590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.531872 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.531878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:57.532053 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.532083 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:03:57.532772 (+ 689us) client_negotiation.cc:240] Negotiation successful
0927 05:03:57.532918 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":411,"threads_started":1}
I20260927 05:03:57.533109 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3755a0b902740a4b55ca55e72fef346" candidate_uuid: "74238dfbd935469b99c860bb4d809f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828daa3e05a24505a0e752f4afed45b5" is_pre_election: true
I20260927 05:03:57.533653 7396 raft_consensus.cc:2454] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74238dfbd935469b99c860bb4d809f19 in term 0.
I20260927 05:03:57.534096 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3755a0b902740a4b55ca55e72fef346" candidate_uuid: "74238dfbd935469b99c860bb4d809f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97570a35a4a45f983a8f39695fe1290" is_pre_election: true
I20260927 05:03:57.534334 7463 leader_election.cc:302] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74238dfbd935469b99c860bb4d809f19, 828daa3e05a24505a0e752f4afed45b5; no voters:
I20260927 05:03:57.534654 7262 raft_consensus.cc:2454] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74238dfbd935469b99c860bb4d809f19 in term 0.
I20260927 05:03:57.534740 7600 raft_consensus.cc:2781] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:03:57.534874 7600 raft_consensus.cc:482] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:03:57.534999 7600 raft_consensus.cc:3037] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:57.537721 7600 raft_consensus.cc:504] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.538333 7600 leader_election.cc:288] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [CANDIDATE]: Term 1 election: Requested vote from peers c97570a35a4a45f983a8f39695fe1290 (127.20.169.65:42427), 828daa3e05a24505a0e752f4afed45b5 (127.20.169.66:42575)
I20260927 05:03:57.538626 7262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3755a0b902740a4b55ca55e72fef346" candidate_uuid: "74238dfbd935469b99c860bb4d809f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97570a35a4a45f983a8f39695fe1290"
I20260927 05:03:57.538818 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3755a0b902740a4b55ca55e72fef346" candidate_uuid: "74238dfbd935469b99c860bb4d809f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "828daa3e05a24505a0e752f4afed45b5"
I20260927 05:03:57.538854 7262 raft_consensus.cc:3037] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:57.539016 7396 raft_consensus.cc:3037] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:57.542133 7396 raft_consensus.cc:2454] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74238dfbd935469b99c860bb4d809f19 in term 1.
I20260927 05:03:57.542133 7262 raft_consensus.cc:2454] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74238dfbd935469b99c860bb4d809f19 in term 1.
I20260927 05:03:57.542549 7465 leader_election.cc:302] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74238dfbd935469b99c860bb4d809f19, c97570a35a4a45f983a8f39695fe1290; no voters:
I20260927 05:03:57.542873 7600 raft_consensus.cc:2781] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:57.543341 7600 raft_consensus.cc:686] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 1 LEADER]: Becoming Leader. State: Replica: 74238dfbd935469b99c860bb4d809f19, State: Running, Role: LEADER
I20260927 05:03:57.543888 7600 consensus_queue.cc:237] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } }
I20260927 05:03:57.548264 7117 catalog_manager.cc:5495] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74238dfbd935469b99c860bb4d809f19 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "74238dfbd935469b99c860bb4d809f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74238dfbd935469b99c860bb4d809f19" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 41399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 } health_report { overall_health: UNKNOWN } } }
I20260927 05:03:57.569703 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.563861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.20.169.67:41399)
0927 05:03:57.564073 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.564080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.564155 (+ 75us) server_negotiation.cc:407] Connection header received
0927 05:03:57.564436 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.564439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.564525 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.564684 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:03:57.565179 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.565957 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.567353 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.567748 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.567886 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.568499 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:03:57.569073 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:03:57.569257 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.569409 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.569455 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 05:03:57.578092 7308 tablet.cc:2369] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:03:57.579013 7262 raft_consensus.cc:1261] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 1 FOLLOWER]: Refusing update from remote peer 74238dfbd935469b99c860bb4d809f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:57.579047 7396 raft_consensus.cc:1261] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 1 FOLLOWER]: Refusing update from remote peer 74238dfbd935469b99c860bb4d809f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:03:57.579952 7604 consensus_queue.cc:1035] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97570a35a4a45f983a8f39695fe1290" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:57.580336 7600 consensus_queue.cc:1035] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "828daa3e05a24505a0e752f4afed45b5" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:03:57.593030 7608 mvcc.cc:205] Tried to move back new op lower bound from 7333828352311300096 to 7333828352183992320. Current Snapshot: MvccSnapshot[applied={T|T < 7333828352311300096}]
I20260927 05:03:57.609442 7117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:57.609750 7117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:03:57.613801 7117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c5711e4b3674012b0bce65cefb10cf6: Sending DeleteTablet for 3 replicas of tablet c3755a0b902740a4b55ca55e72fef346
I20260927 05:03:57.615036 7242 tablet_service.cc:1514] Processing DeleteTablet for tablet c3755a0b902740a4b55ca55e72fef346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:57 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:35010
I20260927 05:03:57.615166 7376 tablet_service.cc:1514] Processing DeleteTablet for tablet c3755a0b902740a4b55ca55e72fef346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:57 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:51118
I20260927 05:03:57.615821 7510 tablet_service.cc:1514] Processing DeleteTablet for tablet c3755a0b902740a4b55ca55e72fef346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:03:57 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:37854
I20260927 05:03:57.619062 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7179
I20260927 05:03:57.635591 7619 tablet_replica.cc:330] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: stopping tablet replica
I20260927 05:03:57.636080 7619 raft_consensus.cc:2229] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:03:57.636869 7619 raft_consensus.cc:2258] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:57.637687 7618 tablet_replica.cc:330] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: stopping tablet replica
I20260927 05:03:57.638098 7618 raft_consensus.cc:2229] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:57.638592 7618 raft_consensus.cc:2258] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:57.638731 7617 tablet_replica.cc:330] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: stopping tablet replica
I20260927 05:03:57.639139 7617 raft_consensus.cc:2229] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:03:57.639624 7617 raft_consensus.cc:2258] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:03:57.640435 7619 ts_tablet_manager.cc:1905] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:57.640635 7618 ts_tablet_manager.cc:1905] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:57.642597 7617 ts_tablet_manager.cc:1905] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:03:57.647400 7618 ts_tablet_manager.cc:1918] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:57.647637 7618 log.cc:1196] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/c3755a0b902740a4b55ca55e72fef346
I20260927 05:03:57.648154 7619 ts_tablet_manager.cc:1918] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:57.648311 7618 ts_tablet_manager.cc:1939] T c3755a0b902740a4b55ca55e72fef346 P 828daa3e05a24505a0e752f4afed45b5: Deleting consensus metadata
I20260927 05:03:57.648411 7619 log.cc:1196] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/c3755a0b902740a4b55ca55e72fef346
I20260927 05:03:57.649041 7619 ts_tablet_manager.cc:1939] T c3755a0b902740a4b55ca55e72fef346 P 74238dfbd935469b99c860bb4d809f19: Deleting consensus metadata
I20260927 05:03:57.650889 7102 catalog_manager.cc:4841] TS 828daa3e05a24505a0e752f4afed45b5 (127.20.169.66:42575): tablet c3755a0b902740a4b55ca55e72fef346 (table test-table [id=115712c51eac4e54ac9bdc885f3753c0]) successfully deleted
I20260927 05:03:57.650911 7105 catalog_manager.cc:4841] TS 74238dfbd935469b99c860bb4d809f19 (127.20.169.67:41399): tablet c3755a0b902740a4b55ca55e72fef346 (table test-table [id=115712c51eac4e54ac9bdc885f3753c0]) successfully deleted
I20260927 05:03:57.652384 7617 ts_tablet_manager.cc:1918] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:03:57.652549 7617 log.cc:1196] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/c3755a0b902740a4b55ca55e72fef346
Sep 27 05:03:57 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:03:57.653091 7617 ts_tablet_manager.cc:1939] T c3755a0b902740a4b55ca55e72fef346 P c97570a35a4a45f983a8f39695fe1290: Deleting consensus metadata
I20260927 05:03:57.654799 7104 catalog_manager.cc:4841] TS c97570a35a4a45f983a8f39695fe1290 (127.20.169.65:42427): tablet c3755a0b902740a4b55ca55e72fef346 (table test-table [id=115712c51eac4e54ac9bdc885f3753c0]) successfully deleted
I20260927 05:03:57.659678 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.620661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.20.169.65:42427)
0927 05:03:57.620910 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.620918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.632103 (+ 11185us) server_negotiation.cc:407] Connection header received
0927 05:03:57.632623 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.632632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.632734 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.632849 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:57.635523 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.636641 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.640031 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.640661 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.640767 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.653842 (+ 13075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:57.653857 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:57.653871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:57.653909 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:57.656801 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.657569 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.657577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.657580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.657716 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.658132 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.658139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.658146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.658486 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:57.658803 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.659128 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.659351 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:03:57.660363 7302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:57.794647 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7179
I20260927 05:03:57.814512 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7313
Sep 27 05:03:57 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:03:57.832124 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.815458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.20.169.66:42575)
0927 05:03:57.815667 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.815675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.815729 (+ 54us) server_negotiation.cc:407] Connection header received
0927 05:03:57.815821 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.815827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.815928 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.816090 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:57.817420 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.818575 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.819887 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.820539 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.820678 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.825209 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:57.825227 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:57.825248 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:57.825304 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:57.829096 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.829802 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.829811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.829816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.829983 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.830355 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.830360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.830364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.830900 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:57.831225 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.831430 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.831669 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:03:57.833052 7436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:57.929204 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7313
I20260927 05:03:57.948570 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7447
Sep 27 05:03:57 dist-test-slave-39f2 krb5kdc[7073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:03:57.965039 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:57.949437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.20.169.67:41399)
0927 05:03:57.949950 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:57.949954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:57.949968 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:03:57.950038 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:57.950042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:57.950100 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:57.950221 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:57.951528 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.952466 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.953823 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:57.954294 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:57.954437 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:57.958857 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:57.958870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:57.958886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:03:57.958918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:57.962113 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.962848 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.962856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.962858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.963021 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:03:57.963418 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:03:57.963425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:03:57.963434 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:03:57.963768 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:57.964108 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:57.964385 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 05:03:57.964608 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260927 05:03:57.965688 7570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:58.089586 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7447
I20260927 05:03:58.110700 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7086
I20260927 05:03:58.111868 7148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:03:58.224545 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7086
2026-09-27T05:03:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4531 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T05:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:03:58Z Disabled control of system clock
I20260927 05:03:58.303020 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33421
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:03:58.653055 7628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:58.653359 7628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:58.653540 7628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:58.661777 7628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:03:58.661876 7628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:58.661978 7628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:58.662114 7628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:03:58.662186 7628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:03:58.672346 7628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:33421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:58.673766 7628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:58.675720 7628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:58.689119 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:58.689198 7634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:58.689786 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:58.691004 7628 server_base.cc:1034] running on GCE node
I20260927 05:03:58.691890 7628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:58.693508 7628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:58.694805 7628 hybrid_clock.cc:648] HybridClock initialized: now 1790485438694749 us; error 62 us; skew 500 ppm
I20260927 05:03:58.695245 7628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:58.697880 7628 webserver.cc:460] Webserver started at http://127.20.169.126:38511/ using document root <none> and password file <none>
I20260927 05:03:58.698652 7628 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:58.698799 7628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:58.699183 7628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:58.702703 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "e1f3dc87cfc741b0b256936152e1ce5c"
format_stamp: "Formatted at 2026-09-27 05:03:58 on dist-test-slave-39f2"
server_key: "78b22a699c42b538673302849877c245"
server_key_iv: "09682e93b2d831c06f90ce805a26f9a5"
server_key_version: "encryptionkey@0"
I20260927 05:03:58.703693 7628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "e1f3dc87cfc741b0b256936152e1ce5c"
format_stamp: "Formatted at 2026-09-27 05:03:58 on dist-test-slave-39f2"
server_key: "78b22a699c42b538673302849877c245"
server_key_iv: "09682e93b2d831c06f90ce805a26f9a5"
server_key_version: "encryptionkey@0"
I20260927 05:03:58.709797 7628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 05:03:58.713629 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:58.715132 7628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:03:58.715373 7628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "e1f3dc87cfc741b0b256936152e1ce5c"
format_stamp: "Formatted at 2026-09-27 05:03:58 on dist-test-slave-39f2"
server_key: "78b22a699c42b538673302849877c245"
server_key_iv: "09682e93b2d831c06f90ce805a26f9a5"
server_key_version: "encryptionkey@0"
I20260927 05:03:58.715611 7628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:58.734493 7628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:58.735389 7628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:58.735713 7628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:58.756768 7628 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:33421
I20260927 05:03:58.756893 7693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:33421 every 8 connection(s)
I20260927 05:03:58.758241 7628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:03:58.762481 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:03:58.766944 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7628
I20260927 05:03:58.767146 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:03:58.767634 21157 external_mini_cluster.cc:1428] Setting key 52980043b6689f124d1928aeb25de86f
I20260927 05:03:58.778518 7694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c: Bootstrap starting.
I20260927 05:03:58.783208 7694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c: Neither blocks nor log segments found. Creating new log.
I20260927 05:03:58.784744 7694 log.cc:824] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:03:58.785794 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:58.769719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.20.169.126:33421)
0927 05:03:58.770906 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:58.770924 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:58.770985 (+ 61us) server_negotiation.cc:407] Connection header received
0927 05:03:58.773219 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:58.773259 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:58.773978 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:58.774485 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:58.775111 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:58.777392 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:58.779425 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:58.780760 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:58.781022 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:58.782303 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:58.782318 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:58.782340 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:58.783193 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:58.783331 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:58.783376 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:58.783507 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:58.783813 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 05:03:58.784168 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":415,"threads_started":1}
I20260927 05:03:58.788102 7694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c: No bootstrap required, opened a new log
I20260927 05:03:58.796650 7694 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33421 } }
I20260927 05:03:58.797145 7694 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:03:58.797344 7694 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f3dc87cfc741b0b256936152e1ce5c, State: Initialized, Role: FOLLOWER
I20260927 05:03:58.798113 7694 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33421 } }
I20260927 05:03:58.798364 7694 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:03:58.798550 7694 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:03:58.798767 7694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:03:58.801441 7694 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33421 } }
I20260927 05:03:58.802047 7694 leader_election.cc:302] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1f3dc87cfc741b0b256936152e1ce5c; no voters:
I20260927 05:03:58.802737 7694 leader_election.cc:288] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:03:58.802915 7699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:03:58.803720 7699 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [term 1 LEADER]: Becoming Leader. State: Replica: e1f3dc87cfc741b0b256936152e1ce5c, State: Running, Role: LEADER
I20260927 05:03:58.804426 7699 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33421 } }
I20260927 05:03:58.805198 7694 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:03:58.807595 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33421 } } }
I20260927 05:03:58.807643 7701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1f3dc87cfc741b0b256936152e1ce5c. Latest consensus state: current_term: 1 leader_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f3dc87cfc741b0b256936152e1ce5c" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33421 } } }
I20260927 05:03:58.808174 7701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:58.808075 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c [sys.catalog]: This master's current role is: LEADER
I20260927 05:03:58.809219 7704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:03:58.815550 7704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:03:58.825407 7704 catalog_manager.cc:1347] Generated new cluster ID: 679579ac262b4ee7add1806cdc037297
I20260927 05:03:58.825510 7704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:03:58.841352 7704 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:03:58.842888 7704 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:03:58.850118 7704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c: Generated new TSK 0
I20260927 05:03:58.850956 7704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:03:58.859103 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33421
--builtin_ntp_servers=127.20.169.84:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:03:59.223582 7718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:59.223893 7718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:59.224044 7718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:59.231405 7718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:59.231530 7718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:59.231698 7718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:03:59.240783 7718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:59.242116 7718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:59.243865 7718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:59.257268 7726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:59.257426 7723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:59.257401 7724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:59.258224 7718 server_base.cc:1034] running on GCE node
I20260927 05:03:59.258769 7718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:59.259694 7718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:59.260948 7718 hybrid_clock.cc:648] HybridClock initialized: now 1790485439260872 us; error 48 us; skew 500 ppm
I20260927 05:03:59.261288 7718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:59.263886 7718 webserver.cc:460] Webserver started at http://127.20.169.65:39823/ using document root <none> and password file <none>
I20260927 05:03:59.264705 7718 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:59.264878 7718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:59.265281 7718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:59.268527 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "6aac508e9fcd4850968b9370ea7e1bc2"
format_stamp: "Formatted at 2026-09-27 05:03:59 on dist-test-slave-39f2"
server_key: "4c6f38e1b52c45c9c43f1fe989ec9d43"
server_key_iv: "2b1cde23f9df8df0f7fc0fd80c3f939f"
server_key_version: "encryptionkey@0"
I20260927 05:03:59.269541 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "6aac508e9fcd4850968b9370ea7e1bc2"
format_stamp: "Formatted at 2026-09-27 05:03:59 on dist-test-slave-39f2"
server_key: "4c6f38e1b52c45c9c43f1fe989ec9d43"
server_key_iv: "2b1cde23f9df8df0f7fc0fd80c3f939f"
server_key_version: "encryptionkey@0"
I20260927 05:03:59.275204 7718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:03:59.279155 7732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:59.280517 7718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:03:59.280694 7718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "6aac508e9fcd4850968b9370ea7e1bc2"
format_stamp: "Formatted at 2026-09-27 05:03:59 on dist-test-slave-39f2"
server_key: "4c6f38e1b52c45c9c43f1fe989ec9d43"
server_key_iv: "2b1cde23f9df8df0f7fc0fd80c3f939f"
server_key_version: "encryptionkey@0"
I20260927 05:03:59.280966 7718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:59.314738 7718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:59.315646 7718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:59.315958 7718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:59.317085 7718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:59.318655 7718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:59.318794 7718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:59.318928 7718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:59.319018 7718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:59.353544 7718 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:42947
I20260927 05:03:59.353632 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:42947 every 8 connection(s)
I20260927 05:03:59.354933 7718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:03:59.361480 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7718
I20260927 05:03:59.361662 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:03:59.362227 21157 external_mini_cluster.cc:1428] Setting key 664512cb9f066fe3ee1535c3a3c6b769
I20260927 05:03:59.365288 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33421
--builtin_ntp_servers=127.20.169.84:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:03:59.369674 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:59.357720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:54853 (local address 127.20.169.126:33421)
0927 05:03:59.358150 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:59.358155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:59.359435 (+ 1280us) server_negotiation.cc:407] Connection header received
0927 05:03:59.361353 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:59.361381 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:59.361489 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:59.361622 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:59.363082 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.363879 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.366358 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.366836 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.366943 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:59.368635 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:59.368644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:59.368652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:59.368677 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:59.368731 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:59.368748 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:59.368889 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:59.369124 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 05:03:59.369238 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":228,"threads_started":1}
I20260927 05:03:59.371153 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:59.358345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33421 (local address 127.20.169.65:54853)
0927 05:03:59.359263 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:59.359300 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:59.361072 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:59.361926 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:59.361941 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:59.362061 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:59.362883 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:59.362905 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.364073 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.364076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:59.366147 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:59.366170 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.367072 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.367080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:59.367409 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:59.368242 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:03:59.368248 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:59.368300 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:03:59.368413 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:59.368886 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:59.368899 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:59.368910 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:03:59.369093 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 05:03:59.369524 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":260,"threads_started":1}
I20260927 05:03:59.372929 7845 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33421
I20260927 05:03:59.373276 7845 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:59.374286 7845 heartbeater.cc:511] Master 127.20.169.126:33421 requested a full tablet report, sending...
I20260927 05:03:59.376569 7659 ts_manager.cc:194] Registered new tserver with Master: 6aac508e9fcd4850968b9370ea7e1bc2 (127.20.169.65:42947)
I20260927 05:03:59.378067 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.65:54853
W20260927 05:03:59.704109 7849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:03:59.704387 7849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:03:59.704531 7849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:03:59.712276 7849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:03:59.712392 7849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:03:59.712555 7849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:03:59.722182 7849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:03:59.723474 7849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:03:59.725370 7849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:03:59.738718 7854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:59.738680 7857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:03:59.738775 7855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:03:59.739717 7849 server_base.cc:1034] running on GCE node
I20260927 05:03:59.740439 7849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:03:59.741524 7849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:03:59.742815 7849 hybrid_clock.cc:648] HybridClock initialized: now 1790485439742743 us; error 51 us; skew 500 ppm
I20260927 05:03:59.743165 7849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:03:59.745581 7849 webserver.cc:460] Webserver started at http://127.20.169.66:42673/ using document root <none> and password file <none>
I20260927 05:03:59.746302 7849 fs_manager.cc:359] Metadata directory not provided
I20260927 05:03:59.746444 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:03:59.746788 7849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:03:59.749929 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "d18be0f8af2243e59ed0e175c44b6f19"
format_stamp: "Formatted at 2026-09-27 05:03:59 on dist-test-slave-39f2"
server_key: "49a0b194e8532d41a998f32cac6fb002"
server_key_iv: "83052e05e4fec764c59e3d24c978d8da"
server_key_version: "encryptionkey@0"
I20260927 05:03:59.750784 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "d18be0f8af2243e59ed0e175c44b6f19"
format_stamp: "Formatted at 2026-09-27 05:03:59 on dist-test-slave-39f2"
server_key: "49a0b194e8532d41a998f32cac6fb002"
server_key_iv: "83052e05e4fec764c59e3d24c978d8da"
server_key_version: "encryptionkey@0"
I20260927 05:03:59.756199 7849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:03:59.759950 7863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:59.761375 7849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:03:59.761554 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "d18be0f8af2243e59ed0e175c44b6f19"
format_stamp: "Formatted at 2026-09-27 05:03:59 on dist-test-slave-39f2"
server_key: "49a0b194e8532d41a998f32cac6fb002"
server_key_iv: "83052e05e4fec764c59e3d24c978d8da"
server_key_version: "encryptionkey@0"
I20260927 05:03:59.761803 7849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:03:59.776046 7849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:03:59.776927 7849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:03:59.777243 7849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:03:59.778213 7849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:03:59.779811 7849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:03:59.779943 7849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:59.780087 7849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:03:59.780184 7849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:03:59.816933 7849 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:32845
I20260927 05:03:59.817003 7975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:32845 every 8 connection(s)
I20260927 05:03:59.818559 7849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:03:59.826429 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7849
I20260927 05:03:59.826606 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:03:59.827176 21157 external_mini_cluster.cc:1428] Setting key 638a9bbec279076b83b2d90686459a28
I20260927 05:03:59.830578 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33421
--builtin_ntp_servers=127.20.169.84:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:03:59.832897 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:59.821669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:47751 (local address 127.20.169.126:33421)
0927 05:03:59.821908 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:03:59.821914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:03:59.823305 (+ 1391us) server_negotiation.cc:407] Connection header received
0927 05:03:59.825077 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:03:59.825082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:03:59.825148 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:03:59.825246 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:03:59.826343 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.827487 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.829601 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.830266 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.830417 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:59.831918 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:03:59.831926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:03:59.831933 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:03:59.831958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:03:59.832000 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:03:59.832018 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:03:59.832111 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:03:59.832349 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 05:03:59.832461 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 05:03:59.833856 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:03:59.822207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33421 (local address 127.20.169.66:47751)
0927 05:03:59.823140 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 05:03:59.823176 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:03:59.824747 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:03:59.825443 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:03:59.825450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:03:59.825535 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:03:59.826167 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:59.826184 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.827663 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.827668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:59.829453 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:03:59.829466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:03:59.830500 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:03:59.830505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:03:59.830772 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:03:59.831582 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:03:59.831585 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:03:59.831627 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:03:59.831709 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:03:59.832118 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:03:59.832127 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:03:59.832133 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:03:59.832329 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 05:03:59.832636 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":285,"threads_started":1}
I20260927 05:03:59.835474 7976 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33421
I20260927 05:03:59.835831 7976 heartbeater.cc:464] Registering TS with master...
I20260927 05:03:59.836558 7976 heartbeater.cc:511] Master 127.20.169.126:33421 requested a full tablet report, sending...
I20260927 05:03:59.838038 7659 ts_manager.cc:194] Registered new tserver with Master: d18be0f8af2243e59ed0e175c44b6f19 (127.20.169.66:32845)
I20260927 05:03:59.838898 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.66:47751
W20260927 05:04:00.161264 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:00.161542 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:00.161692 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:00.169246 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:00.169346 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:00.169512 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:04:00.178555 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:33421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:00.179910 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:00.181667 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:00.194767 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:00.194849 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:00.195058 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:00.195866 7979 server_base.cc:1034] running on GCE node
I20260927 05:04:00.196466 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:00.197484 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:00.198745 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790485440198696 us; error 55 us; skew 500 ppm
I20260927 05:04:00.199101 7979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:04:00.201886 7979 webserver.cc:460] Webserver started at http://127.20.169.67:43029/ using document root <none> and password file <none>
I20260927 05:04:00.202641 7979 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:00.202792 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:00.203140 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:00.206956 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "5df636d4fe27457d9ccb41a4702a1ed2"
format_stamp: "Formatted at 2026-09-27 05:04:00 on dist-test-slave-39f2"
server_key: "2c571df6d1e47ef34774d06a4e83d949"
server_key_iv: "91fcdc4da6e9ae4cd5ef93c5fc231956"
server_key_version: "encryptionkey@0"
I20260927 05:04:00.207840 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "5df636d4fe27457d9ccb41a4702a1ed2"
format_stamp: "Formatted at 2026-09-27 05:04:00 on dist-test-slave-39f2"
server_key: "2c571df6d1e47ef34774d06a4e83d949"
server_key_iv: "91fcdc4da6e9ae4cd5ef93c5fc231956"
server_key_version: "encryptionkey@0"
I20260927 05:04:00.213315 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:04:00.216907 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:00.218345 7979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:00.218538 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "5df636d4fe27457d9ccb41a4702a1ed2"
format_stamp: "Formatted at 2026-09-27 05:04:00 on dist-test-slave-39f2"
server_key: "2c571df6d1e47ef34774d06a4e83d949"
server_key_iv: "91fcdc4da6e9ae4cd5ef93c5fc231956"
server_key_version: "encryptionkey@0"
I20260927 05:04:00.218763 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:00.242084 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:00.242897 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:00.243245 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:00.244263 7979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:00.245831 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:00.245954 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:00.246086 7979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:00.246182 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:00.279531 7979 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:45361
I20260927 05:04:00.279615 8105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:45361 every 8 connection(s)
I20260927 05:04:00.280872 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:04:00.287201 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 7979
I20260927 05:04:00.287388 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:04:00.287835 21157 external_mini_cluster.cc:1428] Setting key 067d37dcfbce54d96d5efa4064a9f363
I20260927 05:04:00.294057 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.283476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:40705 (local address 127.20.169.126:33421)
0927 05:04:00.283655 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.283660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.285225 (+ 1565us) server_negotiation.cc:407] Connection header received
0927 05:04:00.287031 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.287037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.287132 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.287242 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:00.288503 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.289632 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.291625 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.292011 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.292106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.293400 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:00.293404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:00.293408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:00.293425 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:00.293455 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:00.293467 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:00.293537 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.293729 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.293802 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:04:00.295028 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.283954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:33421 (local address 127.20.169.67:40705)
0927 05:04:00.285052 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:00.285100 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:00.286760 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:00.287496 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:00.287508 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:00.287625 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:00.288273 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.288310 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.289805 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.289811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.291496 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.291509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.292189 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.292193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.292423 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.293108 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:04:00.293111 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:00.293147 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:04:00.293225 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:00.293549 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:00.293556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:00.293563 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:04:00.293699 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 05:04:00.293977 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":318,"threads_started":1}
I20260927 05:04:00.296427 8106 heartbeater.cc:347] Connected to a master server at 127.20.169.126:33421
I20260927 05:04:00.296732 8106 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:00.297510 8106 heartbeater.cc:511] Master 127.20.169.126:33421 requested a full tablet report, sending...
I20260927 05:04:00.298979 7659 ts_manager.cc:194] Registered new tserver with Master: 5df636d4fe27457d9ccb41a4702a1ed2 (127.20.169.67:45361)
I20260927 05:04:00.299940 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.67:40705
I20260927 05:04:00.303345 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:04:00.314210 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.307772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.20.169.126:33421)
0927 05:04:00.308010 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.308018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.308569 (+ 551us) server_negotiation.cc:407] Connection header received
0927 05:04:00.308774 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.308779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.308898 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.309008 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:00.309499 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.310202 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.311619 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.312152 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.312286 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.313449 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:00.313462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:00.313466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:00.313483 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:00.313513 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:00.313526 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:00.313676 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.313886 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.313949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 05:04:00.317934 7659 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:04:00.325135 7659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:04:00.350152 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.337130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:32845 (local address 127.0.0.1:51392)
0927 05:04:00.339570 (+ 2440us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:00.339593 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:00.339731 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:00.341141 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:00.341145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:00.341165 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:00.341642 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.341655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.343749 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.343753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.346624 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.346636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.348903 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.348915 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.349249 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.349417 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 05:04:00.349549 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:04:00.349739 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2258,"mutex_wait_us":357,"thread_start_us":2806,"threads_started":1}
I20260927 05:04:00.350193 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.337173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.20.169.65:42947)
0927 05:04:00.340230 (+ 3057us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.340241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.340275 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:04:00.340386 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.340394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.340640 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.341784 (+ 1144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:00.342032 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.343569 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.346884 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.348347 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.348482 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.348718 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.349692 (+ 974us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.349869 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2903,"thread_start_us":2804,"threads_started":1}
I20260927 05:04:00.350872 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.336925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55638)
0927 05:04:00.337830 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:00.337879 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:00.338110 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:00.340913 (+ 2803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:00.340920 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:00.340954 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:00.341530 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.341550 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.343741 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.343746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.346620 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.346636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.348932 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.348939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.349280 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.349416 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 05:04:00.349549 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:04:00.350469 (+ 920us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":1001,"thread_start_us":399,"threads_started":1}
I20260927 05:04:00.351399 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.337506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.20.169.66:32845)
0927 05:04:00.340542 (+ 3036us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.340550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.340576 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:04:00.340671 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.340679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.340913 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.341798 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:00.342032 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.343612 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.346884 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.348347 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.348483 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.348718 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.350835 (+ 2117us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.351030 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2905,"thread_start_us":2789,"threads_started":1}
I20260927 05:04:00.355619 7780 tablet_service.cc:1467] Processing CreateTablet for tablet b0145f6a2b4e41f6a8bee1e949069078 (DEFAULT_TABLE table=test-table [id=bbf23ff2de1f4eed926fdb3638ac073a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:00.355612 7911 tablet_service.cc:1467] Processing CreateTablet for tablet b0145f6a2b4e41f6a8bee1e949069078 (DEFAULT_TABLE table=test-table [id=bbf23ff2de1f4eed926fdb3638ac073a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:00.357131 7780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0145f6a2b4e41f6a8bee1e949069078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:00.357227 7911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0145f6a2b4e41f6a8bee1e949069078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:00.358845 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.339209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:45361 (local address 127.0.0.1:33072)
0927 05:04:00.350410 (+ 11201us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:00.350427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:00.350521 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:00.352134 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:00.352140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:00.352165 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:00.352325 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.352337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.354527 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.354533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.356330 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.356344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.358114 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.358122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.358325 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.358363 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:04:00.358434 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 05:04:00.358566 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11108,"thread_start_us":12299,"threads_started":1}
I20260927 05:04:00.359963 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.339543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.20.169.67:45361)
0927 05:04:00.351526 (+ 11983us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.351534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.351570 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:04:00.351664 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.351673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.351923 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.352499 (+ 576us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:00.352856 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.354375 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.356528 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.357923 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.359090 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.359377 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.359454 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.359615 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11789,"thread_start_us":11631,"threads_started":1}
I20260927 05:04:00.363827 8041 tablet_service.cc:1467] Processing CreateTablet for tablet b0145f6a2b4e41f6a8bee1e949069078 (DEFAULT_TABLE table=test-table [id=bbf23ff2de1f4eed926fdb3638ac073a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:00.365262 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0145f6a2b4e41f6a8bee1e949069078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:00.372400 8124 tablet_bootstrap.cc:492] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Bootstrap starting.
I20260927 05:04:00.375589 8125 tablet_bootstrap.cc:492] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Bootstrap starting.
I20260927 05:04:00.376439 8124 tablet_bootstrap.cc:654] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:00.377830 8124 log.cc:824] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:00.378113 8126 tablet_bootstrap.cc:492] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Bootstrap starting.
I20260927 05:04:00.379002 8125 tablet_bootstrap.cc:654] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:00.380542 8125 log.cc:824] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:00.381060 7845 heartbeater.cc:503] Master 127.20.169.126:33421 was elected leader, sending a full tablet report...
I20260927 05:04:00.381598 8124 tablet_bootstrap.cc:492] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: No bootstrap required, opened a new log
I20260927 05:04:00.381707 8126 tablet_bootstrap.cc:654] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:00.381927 8124 ts_tablet_manager.cc:1397] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260927 05:04:00.383047 8126 log.cc:824] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:00.389950 8126 tablet_bootstrap.cc:492] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: No bootstrap required, opened a new log
I20260927 05:04:00.390302 8126 ts_tablet_manager.cc:1397] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260927 05:04:00.390897 8125 tablet_bootstrap.cc:492] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: No bootstrap required, opened a new log
I20260927 05:04:00.391193 8125 ts_tablet_manager.cc:1397] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260927 05:04:00.394189 8124 raft_consensus.cc:348] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.394675 8124 raft_consensus.cc:374] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:00.394827 8124 raft_consensus.cc:729] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aac508e9fcd4850968b9370ea7e1bc2, State: Initialized, Role: FOLLOWER
I20260927 05:04:00.395466 8124 consensus_queue.cc:260] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.397329 8124 ts_tablet_manager.cc:1428] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 05:04:00.401153 8126 raft_consensus.cc:348] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.401650 8126 raft_consensus.cc:374] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:00.401763 8126 raft_consensus.cc:729] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df636d4fe27457d9ccb41a4702a1ed2, State: Initialized, Role: FOLLOWER
I20260927 05:04:00.401522 8125 raft_consensus.cc:348] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.401980 8125 raft_consensus.cc:374] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:00.402087 8125 raft_consensus.cc:729] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18be0f8af2243e59ed0e175c44b6f19, State: Initialized, Role: FOLLOWER
I20260927 05:04:00.402447 8126 consensus_queue.cc:260] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.402637 8125 consensus_queue.cc:260] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.403645 8106 heartbeater.cc:503] Master 127.20.169.126:33421 was elected leader, sending a full tablet report...
I20260927 05:04:00.404366 8126 ts_tablet_manager.cc:1428] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 05:04:00.404356 8125 ts_tablet_manager.cc:1428] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 05:04:00.405021 7976 heartbeater.cc:503] Master 127.20.169.126:33421 was elected leader, sending a full tablet report...
W20260927 05:04:00.532596 8107 tablet.cc:2369] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:04:00.570989 7977 tablet.cc:2369] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:04:00.607401 7846 tablet.cc:2369] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:00.695533 8131 raft_consensus.cc:482] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:04:00.695881 8131 raft_consensus.cc:504] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.697749 8131 leader_election.cc:288] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aac508e9fcd4850968b9370ea7e1bc2 (127.20.169.65:42947), d18be0f8af2243e59ed0e175c44b6f19 (127.20.169.66:32845)
I20260927 05:04:00.705796 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.698286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:56079 (local address 127.20.169.65:42947)
0927 05:04:00.698528 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.698537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.698564 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:04:00.698846 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.698852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.698933 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.699071 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:00.699473 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.700508 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.703351 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.704637 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.704843 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.704928 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.705391 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.705515 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 05:04:00.705854 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.698412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:43715 (local address 127.20.169.66:32845)
0927 05:04:00.698618 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.698625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.699030 (+ 405us) server_negotiation.cc:407] Connection header received
0927 05:04:00.699222 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.699231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.699335 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.699455 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:00.699962 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.700992 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.703511 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.704753 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.704947 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.705017 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.705384 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.705515 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:04:00.705983 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.698135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.20.169.67:56079)
0927 05:04:00.698468 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:00.698488 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:00.698647 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:00.699112 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:00.699120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:00.699150 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:00.699321 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.699329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.700682 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.700687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.703144 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.703162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.704874 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.704884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.705091 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.705229 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 05:04:00.705343 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 05:04:00.705525 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":163}
I20260927 05:04:00.706034 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.698166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:32845 (local address 127.20.169.67:43715)
0927 05:04:00.698928 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:00.698941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:00.699073 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:00.699493 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:00.699496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:00.699518 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:00.699806 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.699813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.701178 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.701183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.703308 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:00.703324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.704939 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.704944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:00.705228 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.705254 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:04:00.705342 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 05:04:00.705525 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":69,"thread_start_us":445,"threads_started":1}
I20260927 05:04:00.707211 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0145f6a2b4e41f6a8bee1e949069078" candidate_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" is_pre_election: true
I20260927 05:04:00.707222 7931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0145f6a2b4e41f6a8bee1e949069078" candidate_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18be0f8af2243e59ed0e175c44b6f19" is_pre_election: true
I20260927 05:04:00.707728 7800 raft_consensus.cc:2454] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5df636d4fe27457d9ccb41a4702a1ed2 in term 0.
I20260927 05:04:00.707736 7931 raft_consensus.cc:2454] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5df636d4fe27457d9ccb41a4702a1ed2 in term 0.
I20260927 05:04:00.708449 7994 leader_election.cc:302] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5df636d4fe27457d9ccb41a4702a1ed2, 6aac508e9fcd4850968b9370ea7e1bc2, d18be0f8af2243e59ed0e175c44b6f19; no voters:
I20260927 05:04:00.708911 8131 raft_consensus.cc:2781] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:04:00.709057 8131 raft_consensus.cc:482] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:04:00.709164 8131 raft_consensus.cc:3037] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:00.711726 8131 raft_consensus.cc:504] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.712489 8131 leader_election.cc:288] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [CANDIDATE]: Term 1 election: Requested vote from peers 6aac508e9fcd4850968b9370ea7e1bc2 (127.20.169.65:42947), d18be0f8af2243e59ed0e175c44b6f19 (127.20.169.66:32845)
I20260927 05:04:00.712899 7800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0145f6a2b4e41f6a8bee1e949069078" candidate_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aac508e9fcd4850968b9370ea7e1bc2"
I20260927 05:04:00.712980 7931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0145f6a2b4e41f6a8bee1e949069078" candidate_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18be0f8af2243e59ed0e175c44b6f19"
I20260927 05:04:00.713174 7931 raft_consensus.cc:3037] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:00.713167 7800 raft_consensus.cc:3037] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:00.715376 7931 raft_consensus.cc:2454] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5df636d4fe27457d9ccb41a4702a1ed2 in term 1.
I20260927 05:04:00.715519 7800 raft_consensus.cc:2454] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5df636d4fe27457d9ccb41a4702a1ed2 in term 1.
I20260927 05:04:00.715773 7994 leader_election.cc:302] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df636d4fe27457d9ccb41a4702a1ed2, d18be0f8af2243e59ed0e175c44b6f19; no voters:
I20260927 05:04:00.716099 8131 raft_consensus.cc:2781] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:00.716544 8131 raft_consensus.cc:686] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 1 LEADER]: Becoming Leader. State: Replica: 5df636d4fe27457d9ccb41a4702a1ed2, State: Running, Role: LEADER
I20260927 05:04:00.717163 8131 consensus_queue.cc:237] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } }
I20260927 05:04:00.722473 7658 catalog_manager.cc:5495] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5df636d4fe27457d9ccb41a4702a1ed2 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df636d4fe27457d9ccb41a4702a1ed2" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 45361 } health_report { overall_health: HEALTHY } } }
I20260927 05:04:00.828145 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.820493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.20.169.67:45361)
0927 05:04:00.820787 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.820797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.821256 (+ 459us) server_negotiation.cc:407] Connection header received
0927 05:04:00.821473 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.821477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.821560 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.821692 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:00.822533 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.823503 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.825748 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.826162 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.826260 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.826932 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:00.827467 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:00.827619 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.827810 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.827869 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:04:00.837055 7800 raft_consensus.cc:1261] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 1 FOLLOWER]: Refusing update from remote peer 5df636d4fe27457d9ccb41a4702a1ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:00.837055 7931 raft_consensus.cc:1261] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 1 FOLLOWER]: Refusing update from remote peer 5df636d4fe27457d9ccb41a4702a1ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:00.838201 8134 consensus_queue.cc:1035] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aac508e9fcd4850968b9370ea7e1bc2" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 42947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:00.838660 8131 consensus_queue.cc:1035] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18be0f8af2243e59ed0e175c44b6f19" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 32845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:00.851452 8140 mvcc.cc:205] Tried to move back new op lower bound from 7333828365657174016 to 7333828365181984768. Current Snapshot: MvccSnapshot[applied={T|T < 7333828365657174016}]
I20260927 05:04:00.852350 8138 mvcc.cc:205] Tried to move back new op lower bound from 7333828365657174016 to 7333828365181984768. Current Snapshot: MvccSnapshot[applied={T|T < 7333828365657174016}]
I20260927 05:04:00.865772 7658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:04:00.866106 7658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:04:00.869973 7658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f3dc87cfc741b0b256936152e1ce5c: Sending DeleteTablet for 3 replicas of tablet b0145f6a2b4e41f6a8bee1e949069078
I20260927 05:04:00.871372 7911 tablet_service.cc:1514] Processing DeleteTablet for tablet b0145f6a2b4e41f6a8bee1e949069078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:00 UTC) from {username='slave'} at 127.0.0.1:51392
I20260927 05:04:00.871428 7780 tablet_service.cc:1514] Processing DeleteTablet for tablet b0145f6a2b4e41f6a8bee1e949069078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:00 UTC) from {username='slave'} at 127.0.0.1:55638
I20260927 05:04:00.871461 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet b0145f6a2b4e41f6a8bee1e949069078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:00 UTC) from {username='slave'} at 127.0.0.1:33072
I20260927 05:04:00.873781 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7718
I20260927 05:04:00.886880 8150 tablet_replica.cc:330] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: stopping tablet replica
I20260927 05:04:00.887429 8150 raft_consensus.cc:2229] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:04:00.888463 8150 raft_consensus.cc:2258] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:00.891503 8150 ts_tablet_manager.cc:1905] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:00.901279 8150 ts_tablet_manager.cc:1918] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:00.901597 8150 log.cc:1196] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/b0145f6a2b4e41f6a8bee1e949069078
I20260927 05:04:00.902338 8150 ts_tablet_manager.cc:1939] T b0145f6a2b4e41f6a8bee1e949069078 P 5df636d4fe27457d9ccb41a4702a1ed2: Deleting consensus metadata
I20260927 05:04:00.904752 7646 catalog_manager.cc:4841] TS 5df636d4fe27457d9ccb41a4702a1ed2 (127.20.169.67:45361): tablet b0145f6a2b4e41f6a8bee1e949069078 (table test-table [id=bbf23ff2de1f4eed926fdb3638ac073a]) successfully deleted
I20260927 05:04:00.907972 8149 tablet_replica.cc:330] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: stopping tablet replica
I20260927 05:04:00.908576 8148 tablet_replica.cc:330] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: stopping tablet replica
I20260927 05:04:00.909248 8149 raft_consensus.cc:2229] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:04:00.909510 8148 raft_consensus.cc:2229] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:04:00.909893 8149 raft_consensus.cc:2258] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:00.910110 8148 raft_consensus.cc:2258] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:00.911512 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:00.875708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.20.169.65:42947)
0927 05:04:00.875971 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:00.875981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:00.901609 (+ 25628us) server_negotiation.cc:407] Connection header received
0927 05:04:00.902232 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:00.902254 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:00.902391 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:00.902488 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:00.903270 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.904333 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.906756 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:00.907730 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:00.907878 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:00.909557 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:00.909571 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:00.909590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:00.910461 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:00.910530 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:00.910560 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:00.910702 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:00.911008 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 05:04:00.911137 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:04:00.912261 7840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:00.914584 8148 ts_tablet_manager.cc:1905] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:00.921077 8148 ts_tablet_manager.cc:1918] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:00.921260 8148 log.cc:1196] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/b0145f6a2b4e41f6a8bee1e949069078
I20260927 05:04:00.921751 8148 ts_tablet_manager.cc:1939] T b0145f6a2b4e41f6a8bee1e949069078 P d18be0f8af2243e59ed0e175c44b6f19: Deleting consensus metadata
I20260927 05:04:00.923653 7643 catalog_manager.cc:4841] TS d18be0f8af2243e59ed0e175c44b6f19 (127.20.169.66:32845): tablet b0145f6a2b4e41f6a8bee1e949069078 (table test-table [id=bbf23ff2de1f4eed926fdb3638ac073a]) successfully deleted
I20260927 05:04:00.914093 8149 ts_tablet_manager.cc:1905] T b0145f6a2b4e41f6a8bee1e949069078 P 6aac508e9fcd4850968b9370ea7e1bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:01.033375 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7718
W20260927 05:04:01.061158 7646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:42947 (error 108)
I20260927 05:04:01.061857 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7849
I20260927 05:04:01.063594 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.062519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55652)
0927 05:04:01.063110 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.063298 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":614,"threads_started":1}
W20260927 05:04:01.063982 7646 catalog_manager.cc:4584] TS 6aac508e9fcd4850968b9370ea7e1bc2 (127.20.169.65:42947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0145f6a2b4e41f6a8bee1e949069078: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
I20260927 05:04:01.070384 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.062706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.20.169.66:32845)
0927 05:04:01.062943 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:01.062949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:01.062971 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:04:01.063189 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:01.063195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:01.063316 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:01.063450 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:01.064009 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:01.064960 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:01.067048 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:01.067496 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:01.067595 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:01.068918 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:01.068928 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:01.068945 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:01.069556 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:01.069604 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:01.069624 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:01.069718 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:01.069896 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 05:04:01.069985 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:04:01.071141 7971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:01.117854 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.117309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55654)
0927 05:04:01.117553 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.117674 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:04:01.119134 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.118627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55664)
0927 05:04:01.118828 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.118922 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:04:01.159684 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7849
I20260927 05:04:01.178385 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7979
I20260927 05:04:01.180341 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.179796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55674)
0927 05:04:01.180067 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.180165 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 05:04:01.181420 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.181030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55686)
0927 05:04:01.181186 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.181273 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:04:01.186651 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.179356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.20.169.67:45361)
0927 05:04:01.179544 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:01.179549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:01.179571 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:04:01.179641 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:01.179647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:01.179743 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:01.179860 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:01.180275 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:01.181421 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:01.182768 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:01.183559 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:01.183726 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:01.184762 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:01.184775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:01.184795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:01.185728 (+ 933us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:01.185829 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:01.185857 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:01.185976 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:01.186126 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 05:04:01.186231 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:04:01.187510 8101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:01.262246 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.261731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55688)
0927 05:04:01.262001 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.262106 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 05:04:01.263366 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.262957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55700)
0927 05:04:01.263162 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.263231 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 05:04:01.305280 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7979
I20260927 05:04:01.325551 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 7628
I20260927 05:04:01.326681 7689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:01.436594 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:01.435744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:42947 (local address 127.0.0.1:55708)
0927 05:04:01.436165 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:01.436315 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:42947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:04:01.436873 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 7628
2026-09-27T05:04:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3212 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:01 dist-test-slave-39f2 krb5kdc[8160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:01 dist-test-slave-39f2 krb5kdc[8160](info): set up 2 sockets
Sep 27 05:04:01 dist-test-slave-39f2 krb5kdc[8160](info): commencing operation
krb5kdc: starting...
W20260927 05:04:02.580752 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:02 dist-test-slave-39f2 krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:02.852618 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:34991
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:03.185926 8173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:03.186205 8173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:03.186426 8173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:03.193629 8173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:03.193722 8173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:03.193809 8173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:03.193894 8173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:03.193974 8173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:03.203532 8173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:34991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:03.204861 8173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:03.206228 8173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:03.218269 8178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:03.218297 8181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:03.218425 8179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:03.219236 8173 server_base.cc:1034] running on GCE node
I20260927 05:04:03.220115 8173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:03.221653 8173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:03.222877 8173 hybrid_clock.cc:648] HybridClock initialized: now 1790485443222825 us; error 38 us; skew 500 ppm
I20260927 05:04:03.223243 8173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:03 dist-test-slave-39f2 krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:03.226658 8173 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:03.228217 8173 webserver.cc:460] Webserver started at http://127.20.169.126:35095/ using document root <none> and password file <none>
I20260927 05:04:03.228981 8173 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:03.229115 8173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:03.229458 8173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:03.232410 8173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "4f7f44efde914bc2a98b9a3202520c21"
format_stamp: "Formatted at 2026-09-27 05:04:03 on dist-test-slave-39f2"
server_key: "e557e211ae65c3327b34ab613cc0131f"
server_key_iv: "1106ff3de808b30f895047b57513dea4"
server_key_version: "encryptionkey@0"
I20260927 05:04:03.233189 8173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "4f7f44efde914bc2a98b9a3202520c21"
format_stamp: "Formatted at 2026-09-27 05:04:03 on dist-test-slave-39f2"
server_key: "e557e211ae65c3327b34ab613cc0131f"
server_key_iv: "1106ff3de808b30f895047b57513dea4"
server_key_version: "encryptionkey@0"
I20260927 05:04:03.238171 8173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 05:04:03.241674 8188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:03.243129 8173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:03.243355 8173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "4f7f44efde914bc2a98b9a3202520c21"
format_stamp: "Formatted at 2026-09-27 05:04:03 on dist-test-slave-39f2"
server_key: "e557e211ae65c3327b34ab613cc0131f"
server_key_iv: "1106ff3de808b30f895047b57513dea4"
server_key_version: "encryptionkey@0"
I20260927 05:04:03.243534 8173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:03.289579 8173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:03.294730 8173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:03.295096 8173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:03.316148 8173 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:34991
I20260927 05:04:03.316174 8239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:34991 every 8 connection(s)
I20260927 05:04:03.317719 8173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:03.321985 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:03.327919 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8173
I20260927 05:04:03.328110 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:03.328694 21157 external_mini_cluster.cc:1428] Setting key cf7dc83b844fe918511e814b16ea3935
I20260927 05:04:03.334236 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21: Bootstrap starting.
I20260927 05:04:03.338199 8240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:03.339532 8240 log.cc:824] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:03.343286 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21: No bootstrap required, opened a new log
Sep 27 05:04:03 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:03.355705 8240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7f44efde914bc2a98b9a3202520c21" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34991 } }
I20260927 05:04:03.356237 8240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:03.356369 8240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7f44efde914bc2a98b9a3202520c21, State: Initialized, Role: FOLLOWER
I20260927 05:04:03.356575 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:03.330823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.20.169.126:34991)
0927 05:04:03.331681 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:03.331691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:03.331752 (+ 61us) server_negotiation.cc:407] Connection header received
0927 05:04:03.333525 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:03.333548 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:03.333963 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:03.334269 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:03.336131 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:03.337513 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:03.339663 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:03.340652 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:03.340912 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:03.346261 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:03.346274 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:03.346305 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:03.346363 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:03.351387 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:03.352168 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:03.352179 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:03.352189 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:03.352429 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:03.352859 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:03.352865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:03.352868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:03.353748 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:03.354109 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:03.354468 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 05:04:03.354989 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":409,"threads_started":1}
I20260927 05:04:03.357149 8240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7f44efde914bc2a98b9a3202520c21" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34991 } }
I20260927 05:04:03.357460 8240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:03.357633 8240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:03.357846 8240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:03.360409 8240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7f44efde914bc2a98b9a3202520c21" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34991 } }
I20260927 05:04:03.361008 8240 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7f44efde914bc2a98b9a3202520c21; no voters:
I20260927 05:04:03.361779 8240 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:03.361903 8245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:03.362596 8245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7f44efde914bc2a98b9a3202520c21, State: Running, Role: LEADER
I20260927 05:04:03.363250 8245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7f44efde914bc2a98b9a3202520c21" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34991 } }
I20260927 05:04:03.364198 8240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:03.366336 8247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7f44efde914bc2a98b9a3202520c21. Latest consensus state: current_term: 1 leader_uuid: "4f7f44efde914bc2a98b9a3202520c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7f44efde914bc2a98b9a3202520c21" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34991 } } }
I20260927 05:04:03.366786 8247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:03.366474 8246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7f44efde914bc2a98b9a3202520c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7f44efde914bc2a98b9a3202520c21" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 34991 } } }
I20260927 05:04:03.367027 8246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:03.368921 8250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:03.376446 8250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:03.386196 8250 catalog_manager.cc:1347] Generated new cluster ID: e8ea5be1b1414af3bdd33a276816c859
I20260927 05:04:03.386405 8250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:03.405277 8250 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:03.406843 8250 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:03.427708 8250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21: Generated new TSK 0
I20260927 05:04:03.428719 8250 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.169.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
Entry for principal kudu/127.20.169.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab.
I20260927 05:04:03.499676 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34991
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:03.835493 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:03.835794 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:03.835958 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:03.843559 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:03.843660 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:03.843838 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:04:03.853950 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.65.keytab
--principal=kudu/127.20.169.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:03.855245 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:03.856590 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:03.871204 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:03.871752 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:03.872207 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:03.872735 8266 server_base.cc:1034] running on GCE node
I20260927 05:04:03.873588 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:03.874747 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:03.876046 8266 hybrid_clock.cc:648] HybridClock initialized: now 1790485443875940 us; error 99 us; skew 500 ppm
I20260927 05:04:03.876380 8266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:03 dist-test-slave-39f2 krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:03.879765 8266 init.cc:377] Logged in from keytab as kudu/127.20.169.65@KRBTEST.COM (short username kudu)
I20260927 05:04:03.881364 8266 webserver.cc:460] Webserver started at http://127.20.169.65:43795/ using document root <none> and password file <none>
I20260927 05:04:03.882099 8266 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:03.882256 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:03.882596 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:03.885787 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "4bea6b1276c548c3bc8cd71f220d641c"
format_stamp: "Formatted at 2026-09-27 05:04:03 on dist-test-slave-39f2"
server_key: "db186b6ce009a9562cc71a721094ee47"
server_key_iv: "ba0d0b01b0e199003487e20dc9693b7d"
server_key_version: "encryptionkey@0"
I20260927 05:04:03.886570 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "4bea6b1276c548c3bc8cd71f220d641c"
format_stamp: "Formatted at 2026-09-27 05:04:03 on dist-test-slave-39f2"
server_key: "db186b6ce009a9562cc71a721094ee47"
server_key_iv: "ba0d0b01b0e199003487e20dc9693b7d"
server_key_version: "encryptionkey@0"
I20260927 05:04:03.891911 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:04:03.895478 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:03.896742 8266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:04:03.896991 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "4bea6b1276c548c3bc8cd71f220d641c"
format_stamp: "Formatted at 2026-09-27 05:04:03 on dist-test-slave-39f2"
server_key: "db186b6ce009a9562cc71a721094ee47"
server_key_iv: "ba0d0b01b0e199003487e20dc9693b7d"
server_key_version: "encryptionkey@0"
I20260927 05:04:03.897205 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:03.931294 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:03.935491 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:03.935783 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:03.936796 8266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:03.938252 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:03.938355 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:03.938505 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:03.938577 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:03.973708 8266 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:38627
I20260927 05:04:03.973801 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:38627 every 8 connection(s)
I20260927 05:04:03.975011 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:04:03.975749 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8266
I20260927 05:04:03.975970 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:04:03.976406 21157 external_mini_cluster.cc:1428] Setting key f1324146ca23837c06ed30583abec46d
Sep 27 05:04:03 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485443, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.65@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:03.996707 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:03.978005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:55067 (local address 127.20.169.126:34991)
0927 05:04:03.978920 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:03.978926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:03.979733 (+ 807us) server_negotiation.cc:407] Connection header received
0927 05:04:03.981635 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:03.981643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:03.981742 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:03.981866 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:03.983382 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:03.984603 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:03.986240 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:03.986963 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:03.987115 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:03.991112 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:03.991117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:03.991120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:03.991153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:03.993731 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:03.994458 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:03.994470 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:03.994473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:03.994592 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:03.995027 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:03.995032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:03.995037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:03.995293 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:03.995454 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:03.996119 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 05:04:03.996345 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":679,"threads_started":1}
I20260927 05:04:03.997658 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:03.978704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34991 (local address 127.20.169.65:55067)
0927 05:04:03.979534 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:03.979580 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:03.981370 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:03.982088 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:03.982096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:03.982636 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:03.983193 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:03.983210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:03.984806 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:03.984850 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:03.986097 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:03.986108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:03.987177 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:03.987182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:03.987422 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:03.988137 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:03.988175 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:03.990890 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:03.993884 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:03.993892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:03.993908 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:03.994328 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:03.994704 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:03.994710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:03.994713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:03.994905 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:03.995425 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:03.995433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:03.995754 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 05:04:03.996012 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 05:04:03.996531 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":334,"threads_started":1}
I20260927 05:04:03.999282 8394 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34991
I20260927 05:04:03.999588 8394 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:04.000334 8394 heartbeater.cc:511] Master 127.20.169.126:34991 requested a full tablet report, sending...
I20260927 05:04:04.002607 8205 ts_manager.cc:194] Registered new tserver with Master: 4bea6b1276c548c3bc8cd71f220d641c (127.20.169.65:38627)
WARNING: no policy specified for kudu/127.20.169.66@KRBTEST.COM; defaulting to no policy
I20260927 05:04:04.004096 8205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.65@KRBTEST.COM'} at 127.20.169.65:55067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
Entry for principal kudu/127.20.169.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab.
I20260927 05:04:04.036137 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34991
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:04.361070 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:04.361343 8400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:04.361544 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:04.368918 8400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:04.369024 8400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:04.369197 8400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:04:04.378990 8400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.66.keytab
--principal=kudu/127.20.169.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:04.380347 8400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:04.381781 8400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:04.394587 8406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:04.394634 8408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:04.395056 8405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:04.395951 8400 server_base.cc:1034] running on GCE node
I20260927 05:04:04.396535 8400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:04.397538 8400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:04.398778 8400 hybrid_clock.cc:648] HybridClock initialized: now 1790485444398722 us; error 50 us; skew 500 ppm
I20260927 05:04:04.399127 8400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:04 dist-test-slave-39f2 krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:04.402400 8400 init.cc:377] Logged in from keytab as kudu/127.20.169.66@KRBTEST.COM (short username kudu)
I20260927 05:04:04.403882 8400 webserver.cc:460] Webserver started at http://127.20.169.66:42899/ using document root <none> and password file <none>
I20260927 05:04:04.404574 8400 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:04.404719 8400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:04.405067 8400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:04.407981 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "76e8f36cea8f45f6b61522882c5aae08"
format_stamp: "Formatted at 2026-09-27 05:04:04 on dist-test-slave-39f2"
server_key: "fc8d7eca15cefb7f4601331da7abc35c"
server_key_iv: "e2fd215c2422fe9adc02142483592e82"
server_key_version: "encryptionkey@0"
I20260927 05:04:04.408764 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "76e8f36cea8f45f6b61522882c5aae08"
format_stamp: "Formatted at 2026-09-27 05:04:04 on dist-test-slave-39f2"
server_key: "fc8d7eca15cefb7f4601331da7abc35c"
server_key_iv: "e2fd215c2422fe9adc02142483592e82"
server_key_version: "encryptionkey@0"
I20260927 05:04:04.414152 8400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:04:04.417757 8415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:04.419106 8400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:04:04.419284 8400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "76e8f36cea8f45f6b61522882c5aae08"
format_stamp: "Formatted at 2026-09-27 05:04:04 on dist-test-slave-39f2"
server_key: "fc8d7eca15cefb7f4601331da7abc35c"
server_key_iv: "e2fd215c2422fe9adc02142483592e82"
server_key_version: "encryptionkey@0"
I20260927 05:04:04.419502 8400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:04.456132 8400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:04.460618 8400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:04.460969 8400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:04.462005 8400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:04.463359 8400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:04.463433 8400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:04.463521 8400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:04.463563 8400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:04.498849 8400 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:38687
I20260927 05:04:04.498948 8527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:38687 every 8 connection(s)
I20260927 05:04:04.500205 8400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:04:04.502067 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8400
I20260927 05:04:04.502249 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:04:04.502861 21157 external_mini_cluster.cc:1428] Setting key d6a754e03fe4d1556c2b19378d81e976
Sep 27 05:04:04 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.66@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:04.523869 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:04.502998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:57019 (local address 127.20.169.126:34991)
0927 05:04:04.503483 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:04.503487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:04.504624 (+ 1137us) server_negotiation.cc:407] Connection header received
0927 05:04:04.506484 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:04.506493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:04.506586 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:04.506709 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:04.508244 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:04.509401 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:04.511634 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:04.512373 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:04.512520 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:04.516585 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:04.516595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:04.516603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:04.516650 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:04.520330 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:04.521201 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:04.521209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:04.521216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:04.521390 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:04.521797 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:04.521804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:04.521810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:04.522201 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:04.522366 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:04.523086 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 05:04:04.523377 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":226,"threads_started":1}
I20260927 05:04:04.524602 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:04.503631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34991 (local address 127.20.169.66:57019)
0927 05:04:04.504454 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:04.504508 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:04.506210 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:04.506950 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:04.506959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:04.507476 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:04.508062 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:04.508080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:04.509586 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:04.509593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:04.511471 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:04.511487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:04.512596 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:04.512604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:04.512934 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:04.513700 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:04.513737 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:04.516351 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:04.520520 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:04.520530 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:04.520549 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:04.521086 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:04.521503 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:04.521506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:04.521508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:04.521677 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:04.522331 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:04.522337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:04.522640 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 05:04:04.522922 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 05:04:04.523404 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":339,"threads_started":1}
I20260927 05:04:04.526087 8528 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34991
I20260927 05:04:04.526425 8528 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:04.527129 8528 heartbeater.cc:511] Master 127.20.169.126:34991 requested a full tablet report, sending...
I20260927 05:04:04.528625 8205 ts_manager.cc:194] Registered new tserver with Master: 76e8f36cea8f45f6b61522882c5aae08 (127.20.169.66:38687)
I20260927 05:04:04.529474 8205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.66@KRBTEST.COM'} at 127.20.169.66:57019
WARNING: no policy specified for kudu/127.20.169.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
Entry for principal kudu/127.20.169.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab.
I20260927 05:04:04.565331 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34991
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:04.897348 8534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:04.897622 8534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:04.897714 8534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:04.904983 8534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:04.905089 8534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:04.905200 8534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:04:04.914860 8534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.67.keytab
--principal=kudu/127.20.169.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:04.916244 8534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:04.917651 8534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:04.930830 8539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:04.930920 8540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:04.931209 8542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:04.931772 8534 server_base.cc:1034] running on GCE node
I20260927 05:04:04.932417 8534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:04.933416 8534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:04.934679 8534 hybrid_clock.cc:648] HybridClock initialized: now 1790485444934623 us; error 36 us; skew 500 ppm
I20260927 05:04:04.935020 8534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:04 dist-test-slave-39f2 krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:04.938534 8534 init.cc:377] Logged in from keytab as kudu/127.20.169.67@KRBTEST.COM (short username kudu)
I20260927 05:04:04.940191 8534 webserver.cc:460] Webserver started at http://127.20.169.67:46011/ using document root <none> and password file <none>
I20260927 05:04:04.940943 8534 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:04.941061 8534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:04.941339 8534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:04.944377 8534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "2c10de56dcf840948db8c8b2cb587b02"
format_stamp: "Formatted at 2026-09-27 05:04:04 on dist-test-slave-39f2"
server_key: "f70fdfae709e741dbc0c4a9c69759b08"
server_key_iv: "ab1eea1bced8a48eb3847d4ba05bfce7"
server_key_version: "encryptionkey@0"
I20260927 05:04:04.945236 8534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "2c10de56dcf840948db8c8b2cb587b02"
format_stamp: "Formatted at 2026-09-27 05:04:04 on dist-test-slave-39f2"
server_key: "f70fdfae709e741dbc0c4a9c69759b08"
server_key_iv: "ab1eea1bced8a48eb3847d4ba05bfce7"
server_key_version: "encryptionkey@0"
I20260927 05:04:04.951182 8534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260927 05:04:04.954991 8549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:04.956377 8534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:04:04.956525 8534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "2c10de56dcf840948db8c8b2cb587b02"
format_stamp: "Formatted at 2026-09-27 05:04:04 on dist-test-slave-39f2"
server_key: "f70fdfae709e741dbc0c4a9c69759b08"
server_key_iv: "ab1eea1bced8a48eb3847d4ba05bfce7"
server_key_version: "encryptionkey@0"
I20260927 05:04:04.956693 8534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:04.981801 8534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:04.986196 8534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:04.986500 8534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:04.987428 8534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:04.988744 8534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:04.988888 8534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:04.989050 8534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:04.989128 8534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:05.007097 8394 heartbeater.cc:503] Master 127.20.169.126:34991 was elected leader, sending a full tablet report...
I20260927 05:04:05.026630 8534 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:43155
I20260927 05:04:05.026723 8661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:43155 every 8 connection(s)
I20260927 05:04:05.028025 8534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:04:05.031271 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8534
I20260927 05:04:05.031466 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:04:05.031956 21157 external_mini_cluster.cc:1428] Setting key dd25f5845ab45e37962660b6435fb122
Sep 27 05:04:05 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485444, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.67@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:05.050349 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.030897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:34343 (local address 127.20.169.126:34991)
0927 05:04:05.031460 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.031468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.032636 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 05:04:05.034807 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.034812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.034929 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.035057 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:05.036650 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.037574 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.039304 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.039739 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.039885 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.043537 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:05.043542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:05.043545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:05.043578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:05.046817 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.047882 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.047886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.047888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.048005 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.048499 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.048503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.048505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.048905 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:05.049036 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.049837 (+ 801us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.050035 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
I20260927 05:04:05.051445 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.031550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:34991 (local address 127.20.169.67:34343)
0927 05:04:05.032440 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:05.032496 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:05.034523 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:05.035312 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:05.035321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:05.035856 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:05.036467 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.036490 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.037745 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.037751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.039142 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.039155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.039977 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.039985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.040239 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.040945 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:05.040969 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:05.043340 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:05.046951 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:05.046962 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:05.046979 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:05.047732 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:05.048108 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:05.048112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:05.048114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:05.048380 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:05.049056 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:05.049062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:05.049467 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 05:04:05.049716 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 05:04:05.050130 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":356,"threads_started":1}
I20260927 05:04:05.052980 8662 heartbeater.cc:347] Connected to a master server at 127.20.169.126:34991
I20260927 05:04:05.053349 8662 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:05.054095 8662 heartbeater.cc:511] Master 127.20.169.126:34991 requested a full tablet report, sending...
I20260927 05:04:05.055619 8205 ts_manager.cc:194] Registered new tserver with Master: 2c10de56dcf840948db8c8b2cb587b02 (127.20.169.67:43155)
I20260927 05:04:05.056485 8205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.169.67@KRBTEST.COM'} at 127.20.169.67:34343
I20260927 05:04:05.060046 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:04:05.078197 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.064561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.20.169.126:34991)
0927 05:04:05.064840 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.064850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.065217 (+ 367us) server_negotiation.cc:407] Connection header received
0927 05:04:05.065394 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.065398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.065496 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.065576 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:05.067083 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.067836 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.069352 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.069745 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.069850 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.072494 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:05.072500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:05.072503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:05.072536 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:05.075453 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.076148 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.076151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.076153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.076269 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.076677 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.076680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.076682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.077062 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:05.077240 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.077641 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.077872 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:04:05.082093 8205 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:04:05.089071 8205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:04:05.129192 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.102542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:43155 (local address 127.0.0.1:41982)
0927 05:04:05.119811 (+ 17269us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:05.119836 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:05.120009 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:05.120864 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:05.120872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:05.120924 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:05.121540 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.121560 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.123529 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.123536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.126452 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.126469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.128127 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.128136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.128316 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.128370 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 05:04:05.128486 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 05:04:05.128676 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17093,"mutex_wait_us":66,"spinlock_wait_cycles":896,"thread_start_us":17410,"threads_started":1}
I20260927 05:04:05.129523 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.103147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.20.169.67:43155)
0927 05:04:05.117152 (+ 14005us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.117166 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.120166 (+ 3000us) server_negotiation.cc:407] Connection header received
0927 05:04:05.120304 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.120312 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.120591 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.120829 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:05.121803 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.123315 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.126655 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.127918 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.128066 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.128336 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.128955 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.129133 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13794,"thread_start_us":13640,"threads_started":1}
I20260927 05:04:05.130198 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.102543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.20.169.66:38687)
0927 05:04:05.120197 (+ 17654us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.120205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.120231 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:04:05.120318 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.120325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.120591 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.120835 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:05.121861 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.123420 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.127194 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.128380 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.128519 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.128703 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.129611 (+ 908us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.129790 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17483,"thread_start_us":17583,"threads_started":1}
I20260927 05:04:05.130528 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.103428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:38627 (local address 127.0.0.1:55762)
0927 05:04:05.121382 (+ 17954us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:05.121397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:05.121587 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:05.122494 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:05.122500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:05.122529 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:05.122930 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.122944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.124792 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.124797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.127273 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.127284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.129568 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.129575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.129788 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.129824 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:04:05.129925 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:04:05.130073 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17832,"mutex_wait_us":272,"thread_start_us":16012,"threads_started":1}
I20260927 05:04:05.130890 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.102313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:38687 (local address 127.0.0.1:51786)
0927 05:04:05.117507 (+ 15194us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:05.117562 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:05.117786 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:05.120864 (+ 3078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:05.120872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:05.120931 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:05.121670 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.121683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.123548 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.123555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.127014 (+ 3459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.127028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.128619 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.128627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.128864 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.128903 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:04:05.128995 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 05:04:05.129168 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14845,"mutex_wait_us":124,"spinlock_wait_cycles":9472,"thread_start_us":15705,"threads_started":1}
I20260927 05:04:05.132438 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.104251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.20.169.65:38627)
0927 05:04:05.119213 (+ 14962us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.119226 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.121525 (+ 2299us) server_negotiation.cc:407] Connection header received
0927 05:04:05.122022 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.122031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.122286 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.122476 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:05.123096 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.124621 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.127445 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.129040 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.129175 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.129409 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.130813 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.132055 (+ 1242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14762,"thread_start_us":14593,"threads_started":1}
I20260927 05:04:05.134029 8597 tablet_service.cc:1467] Processing CreateTablet for tablet 269a99f697f84360a56c3a9a42251c9b (DEFAULT_TABLE table=test-table [id=99d0ed7025684e07af15f633b0602561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:05.134838 8463 tablet_service.cc:1467] Processing CreateTablet for tablet 269a99f697f84360a56c3a9a42251c9b (DEFAULT_TABLE table=test-table [id=99d0ed7025684e07af15f633b0602561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:05.135474 8597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269a99f697f84360a56c3a9a42251c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:05.136225 8463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269a99f697f84360a56c3a9a42251c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:05.136171 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 269a99f697f84360a56c3a9a42251c9b (DEFAULT_TABLE table=test-table [id=99d0ed7025684e07af15f633b0602561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:05.137521 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269a99f697f84360a56c3a9a42251c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:05.147258 8681 tablet_bootstrap.cc:492] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Bootstrap starting.
I20260927 05:04:05.150386 8682 tablet_bootstrap.cc:492] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Bootstrap starting.
I20260927 05:04:05.151157 8681 tablet_bootstrap.cc:654] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:05.151721 8683 tablet_bootstrap.cc:492] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Bootstrap starting.
I20260927 05:04:05.153301 8681 log.cc:824] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:05.154167 8682 tablet_bootstrap.cc:654] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:05.154933 8683 tablet_bootstrap.cc:654] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:05.155424 8682 log.cc:824] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:05.155900 8683 log.cc:824] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:05.156678 8681 tablet_bootstrap.cc:492] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: No bootstrap required, opened a new log
I20260927 05:04:05.157006 8681 ts_tablet_manager.cc:1397] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 05:04:05.158684 8682 tablet_bootstrap.cc:492] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: No bootstrap required, opened a new log
I20260927 05:04:05.158970 8682 ts_tablet_manager.cc:1397] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:04:05.163451 8683 tablet_bootstrap.cc:492] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: No bootstrap required, opened a new log
I20260927 05:04:05.163736 8683 ts_tablet_manager.cc:1397] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260927 05:04:05.168566 8681 raft_consensus.cc:348] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.169073 8681 raft_consensus.cc:374] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:05.169202 8681 raft_consensus.cc:729] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c10de56dcf840948db8c8b2cb587b02, State: Initialized, Role: FOLLOWER
I20260927 05:04:05.169878 8681 consensus_queue.cc:260] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.170213 8682 raft_consensus.cc:348] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.170699 8682 raft_consensus.cc:374] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:05.170830 8682 raft_consensus.cc:729] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e8f36cea8f45f6b61522882c5aae08, State: Initialized, Role: FOLLOWER
I20260927 05:04:05.171404 8662 heartbeater.cc:503] Master 127.20.169.126:34991 was elected leader, sending a full tablet report...
I20260927 05:04:05.171545 8682 consensus_queue.cc:260] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.171999 8681 ts_tablet_manager.cc:1428] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:04:05.173106 8528 heartbeater.cc:503] Master 127.20.169.126:34991 was elected leader, sending a full tablet report...
I20260927 05:04:05.173775 8682 ts_tablet_manager.cc:1428] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 05:04:05.174131 8683 raft_consensus.cc:348] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.174589 8683 raft_consensus.cc:374] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:05.174696 8683 raft_consensus.cc:729] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea6b1276c548c3bc8cd71f220d641c, State: Initialized, Role: FOLLOWER
I20260927 05:04:05.175357 8683 consensus_queue.cc:260] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.177292 8683 ts_tablet_manager.cc:1428] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260927 05:04:05.227587 8395 tablet.cc:2369] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:04:05.252442 8529 tablet.cc:2369] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:05.277657 8687 raft_consensus.cc:482] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:04:05.277948 8687 raft_consensus.cc:504] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.279830 8687 leader_election.cc:288] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76e8f36cea8f45f6b61522882c5aae08 (127.20.169.66:38687), 4bea6b1276c548c3bc8cd71f220d641c (127.20.169.65:38627)
W20260927 05:04:05.279870 8663 tablet.cc:2369] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:05.286914 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.280494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:58589 (local address 127.20.169.66:38687)
0927 05:04:05.280764 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.280774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.280937 (+ 163us) server_negotiation.cc:407] Connection header received
0927 05:04:05.281187 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.281191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.281261 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.281371 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:05.281834 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.282854 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.284934 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.286087 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.286204 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.286257 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.286613 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.286701 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:04:05.287009 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.280111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:38687 (local address 127.20.169.67:58589)
0927 05:04:05.280763 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:05.280791 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:05.280969 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:05.281395 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:05.281400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:05.281433 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:05.281660 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.281673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.283010 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.283017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.284721 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.284735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.286236 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.286243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.286425 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.286477 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 05:04:05.286585 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:04:05.286718 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473}
I20260927 05:04:05.287995 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.280475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:38641 (local address 127.20.169.65:38627)
0927 05:04:05.281019 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.281028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.281670 (+ 642us) server_negotiation.cc:407] Connection header received
0927 05:04:05.281791 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.281796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.281887 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.281962 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:05.282617 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.283772 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.285889 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.287009 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.287166 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.287242 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.287617 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.287740 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260927 05:04:05.288046 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.280323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:38627 (local address 127.20.169.67:38641)
0927 05:04:05.281549 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:05.281563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:05.281675 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:05.282004 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:05.282008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:05.282033 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:05.282417 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.282429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.283952 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.283956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.285707 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:05.285721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.287327 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.287334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:05.287463 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.287485 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:04:05.287591 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 05:04:05.287710 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":902,"threads_started":1}
I20260927 05:04:05.287950 8483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269a99f697f84360a56c3a9a42251c9b" candidate_uuid: "2c10de56dcf840948db8c8b2cb587b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e8f36cea8f45f6b61522882c5aae08" is_pre_election: true
I20260927 05:04:05.288502 8483 raft_consensus.cc:2454] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c10de56dcf840948db8c8b2cb587b02 in term 0.
I20260927 05:04:05.289058 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269a99f697f84360a56c3a9a42251c9b" candidate_uuid: "2c10de56dcf840948db8c8b2cb587b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea6b1276c548c3bc8cd71f220d641c" is_pre_election: true
I20260927 05:04:05.289189 8550 leader_election.cc:302] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c10de56dcf840948db8c8b2cb587b02, 76e8f36cea8f45f6b61522882c5aae08; no voters:
I20260927 05:04:05.289608 8687 raft_consensus.cc:2781] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:04:05.289551 8349 raft_consensus.cc:2454] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c10de56dcf840948db8c8b2cb587b02 in term 0.
I20260927 05:04:05.289762 8687 raft_consensus.cc:482] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:04:05.289884 8687 raft_consensus.cc:3037] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:05.292033 8687 raft_consensus.cc:504] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.292779 8687 leader_election.cc:288] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [CANDIDATE]: Term 1 election: Requested vote from peers 76e8f36cea8f45f6b61522882c5aae08 (127.20.169.66:38687), 4bea6b1276c548c3bc8cd71f220d641c (127.20.169.65:38627)
I20260927 05:04:05.293126 8483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269a99f697f84360a56c3a9a42251c9b" candidate_uuid: "2c10de56dcf840948db8c8b2cb587b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e8f36cea8f45f6b61522882c5aae08"
I20260927 05:04:05.293241 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269a99f697f84360a56c3a9a42251c9b" candidate_uuid: "2c10de56dcf840948db8c8b2cb587b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea6b1276c548c3bc8cd71f220d641c"
I20260927 05:04:05.293350 8483 raft_consensus.cc:3037] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:05.293509 8349 raft_consensus.cc:3037] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:05.295622 8483 raft_consensus.cc:2454] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c10de56dcf840948db8c8b2cb587b02 in term 1.
I20260927 05:04:05.295848 8349 raft_consensus.cc:2454] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c10de56dcf840948db8c8b2cb587b02 in term 1.
I20260927 05:04:05.296021 8550 leader_election.cc:302] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c10de56dcf840948db8c8b2cb587b02, 76e8f36cea8f45f6b61522882c5aae08; no voters:
I20260927 05:04:05.296314 8687 raft_consensus.cc:2781] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:05.296756 8687 raft_consensus.cc:686] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 1 LEADER]: Becoming Leader. State: Replica: 2c10de56dcf840948db8c8b2cb587b02, State: Running, Role: LEADER
I20260927 05:04:05.297314 8687 consensus_queue.cc:237] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } }
I20260927 05:04:05.301644 8203 catalog_manager.cc:5495] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c10de56dcf840948db8c8b2cb587b02 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "2c10de56dcf840948db8c8b2cb587b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c10de56dcf840948db8c8b2cb587b02" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 43155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 } health_report { overall_health: UNKNOWN } } }
I20260927 05:04:05.352586 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.346757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.20.169.67:43155)
0927 05:04:05.347008 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.347013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.347063 (+ 50us) server_negotiation.cc:407] Connection header received
0927 05:04:05.347319 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.347323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.347465 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.347631 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:05.348023 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.348796 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.350379 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.350796 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.350905 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.351394 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:05.351926 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:05.352045 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.352148 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.352215 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:04:05.361470 8349 raft_consensus.cc:1261] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 1 FOLLOWER]: Refusing update from remote peer 2c10de56dcf840948db8c8b2cb587b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:05.361470 8483 raft_consensus.cc:1261] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 1 FOLLOWER]: Refusing update from remote peer 2c10de56dcf840948db8c8b2cb587b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:05.362568 8691 consensus_queue.cc:1035] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e8f36cea8f45f6b61522882c5aae08" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 38687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:05.362972 8687 consensus_queue.cc:1035] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea6b1276c548c3bc8cd71f220d641c" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 38627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:05.375425 8694 mvcc.cc:205] Tried to move back new op lower bound from 7333828384189276160 to 7333828383941951488. Current Snapshot: MvccSnapshot[applied={T|T < 7333828384189276160}]
I20260927 05:04:05.376516 8695 mvcc.cc:205] Tried to move back new op lower bound from 7333828384189276160 to 7333828383941951488. Current Snapshot: MvccSnapshot[applied={T|T < 7333828384189276160}]
I20260927 05:04:05.394954 8205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:04:05.395246 8205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:04:05.399137 8205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f7f44efde914bc2a98b9a3202520c21: Sending DeleteTablet for 3 replicas of tablet 269a99f697f84360a56c3a9a42251c9b
I20260927 05:04:05.400488 8463 tablet_service.cc:1514] Processing DeleteTablet for tablet 269a99f697f84360a56c3a9a42251c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:05 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:51786
I20260927 05:04:05.400499 8597 tablet_service.cc:1514] Processing DeleteTablet for tablet 269a99f697f84360a56c3a9a42251c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:05 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260927 05:04:05.401141 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 269a99f697f84360a56c3a9a42251c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:05 UTC) from {username='kudu', principal='kudu/127.20.169.126@KRBTEST.COM'} at 127.0.0.1:55762
I20260927 05:04:05.402174 8704 tablet_replica.cc:330] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: stopping tablet replica
I20260927 05:04:05.402513 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8266
I20260927 05:04:05.402797 8704 raft_consensus.cc:2229] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:04:05.403652 8704 raft_consensus.cc:2258] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:05.404928 8706 tablet_replica.cc:330] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: stopping tablet replica
I20260927 05:04:05.405367 8706 raft_consensus.cc:2229] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:04:05.405905 8706 raft_consensus.cc:2258] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:05.406648 8705 tablet_replica.cc:330] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: stopping tablet replica
I20260927 05:04:05.407035 8704 ts_tablet_manager.cc:1905] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:05.407125 8705 raft_consensus.cc:2229] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:04:05.407680 8705 raft_consensus.cc:2258] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:05.410189 8705 ts_tablet_manager.cc:1905] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:05.414693 8706 ts_tablet_manager.cc:1905] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:05.416718 8704 ts_tablet_manager.cc:1918] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:05.417037 8704 log.cc:1196] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/269a99f697f84360a56c3a9a42251c9b
I20260927 05:04:05.417276 8705 ts_tablet_manager.cc:1918] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:05.417445 8705 log.cc:1196] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/269a99f697f84360a56c3a9a42251c9b
I20260927 05:04:05.417961 8704 ts_tablet_manager.cc:1939] T 269a99f697f84360a56c3a9a42251c9b P 2c10de56dcf840948db8c8b2cb587b02: Deleting consensus metadata
I20260927 05:04:05.418004 8705 ts_tablet_manager.cc:1939] T 269a99f697f84360a56c3a9a42251c9b P 4bea6b1276c548c3bc8cd71f220d641c: Deleting consensus metadata
I20260927 05:04:05.419906 8192 catalog_manager.cc:4841] TS 4bea6b1276c548c3bc8cd71f220d641c (127.20.169.65:38627): tablet 269a99f697f84360a56c3a9a42251c9b (table test-table [id=99d0ed7025684e07af15f633b0602561]) successfully deleted
I20260927 05:04:05.420066 8190 catalog_manager.cc:4841] TS 2c10de56dcf840948db8c8b2cb587b02 (127.20.169.67:43155): tablet 269a99f697f84360a56c3a9a42251c9b (table test-table [id=99d0ed7025684e07af15f633b0602561]) successfully deleted
I20260927 05:04:05.424547 8706 ts_tablet_manager.cc:1918] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:05.424713 8706 log.cc:1196] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/269a99f697f84360a56c3a9a42251c9b
I20260927 05:04:05.425240 8706 ts_tablet_manager.cc:1939] T 269a99f697f84360a56c3a9a42251c9b P 76e8f36cea8f45f6b61522882c5aae08: Deleting consensus metadata
Sep 27 05:04:05 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.65@KRBTEST.COM
I20260927 05:04:05.427073 8189 catalog_manager.cc:4841] TS 76e8f36cea8f45f6b61522882c5aae08 (127.20.169.66:38687): tablet 269a99f697f84360a56c3a9a42251c9b (table test-table [id=99d0ed7025684e07af15f633b0602561]) successfully deleted
I20260927 05:04:05.433403 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.403628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.20.169.65:38627)
0927 05:04:05.403793 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.403799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.411134 (+ 7335us) server_negotiation.cc:407] Connection header received
0927 05:04:05.411205 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.411210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.411308 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.411424 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:05.417108 (+ 5684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.418369 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.420206 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.420857 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.420983 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.426982 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:05.426996 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:05.427012 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:05.427063 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:05.430093 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.430997 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.431011 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.431013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.431142 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.431593 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.431596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.431598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.431917 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:05.432232 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.432592 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.432901 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:04:05.434048 8389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:05.574847 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8266
I20260927 05:04:05.594118 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8400
Sep 27 05:04:05 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.66@KRBTEST.COM
I20260927 05:04:05.610050 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.594920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.20.169.66:38687)
0927 05:04:05.595105 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.595112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.595137 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:04:05.595386 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.595392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.595469 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.595581 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:05.596921 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.597676 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.598963 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.599364 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.599503 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.604170 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:05.604185 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:05.604200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:05.604238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:05.607157 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.607850 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.607865 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.607867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.608010 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.608410 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.608419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.608421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.608793 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:05.609123 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.609472 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.609689 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:04:05.610772 8523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:05.702566 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8400
I20260927 05:04:05.720944 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8534
Sep 27 05:04:05 dist-test-slave-39f2 krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.67@KRBTEST.COM
I20260927 05:04:05.738991 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:05.721898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.20.169.67:43155)
0927 05:04:05.722133 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:05.722139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:05.722162 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:04:05.722242 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:05.722248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:05.722338 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:05.722452 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:05.723816 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.725186 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.726412 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:05.727080 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:05.727211 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:05.731902 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:05.731918 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:05.731939 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:05.732009 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:05.735946 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.736667 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.736682 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.736686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.736945 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:05.737322 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:05.737328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:05.737332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:05.737806 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:05.738152 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:05.738302 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 05:04:05.738546 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:04:05.739934 8657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:05.861346 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8534
I20260927 05:04:05.885597 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8173
I20260927 05:04:05.886766 8235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:05.997948 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8173
2026-09-27T05:04:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4562 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T05:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:06Z Disabled control of system clock
I20260927 05:04:06.080494 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39223
--webserver_interface=127.20.169.126
--webserver_port=0
--builtin_ntp_servers=127.20.169.84:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:39223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:04:06.415992 8716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:06.416354 8716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:06.416518 8716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:06.424048 8716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:06.424168 8716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:06.424269 8716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:06.424386 8716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:06.424472 8716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:06.433881 8716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:39223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:06.435218 8716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:06.437069 8716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:06.449272 8724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:06.449316 8722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:06.449442 8721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:06.450012 8716 server_base.cc:1034] running on GCE node
I20260927 05:04:06.450769 8716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:06.452126 8716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:06.453366 8716 hybrid_clock.cc:648] HybridClock initialized: now 1790485446453319 us; error 35 us; skew 500 ppm
I20260927 05:04:06.453670 8716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:04:06.455880 8716 webserver.cc:460] Webserver started at http://127.20.169.126:41369/ using document root <none> and password file <none>
I20260927 05:04:06.456506 8716 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:06.456593 8716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:06.456912 8716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:06.459686 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "6b56a79938704b41b2c3a6abf5fd9e31"
format_stamp: "Formatted at 2026-09-27 05:04:06 on dist-test-slave-39f2"
server_key: "acaf68dbdc82d0acdce2ff2720599377"
server_key_iv: "581e6bc6cbc2ec893958887b66f7fe31"
server_key_version: "encryptionkey@0"
I20260927 05:04:06.460426 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "6b56a79938704b41b2c3a6abf5fd9e31"
format_stamp: "Formatted at 2026-09-27 05:04:06 on dist-test-slave-39f2"
server_key: "acaf68dbdc82d0acdce2ff2720599377"
server_key_iv: "581e6bc6cbc2ec893958887b66f7fe31"
server_key_version: "encryptionkey@0"
I20260927 05:04:06.465528 8716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:04:06.469055 8730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:06.470474 8716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:04:06.470620 8716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "6b56a79938704b41b2c3a6abf5fd9e31"
format_stamp: "Formatted at 2026-09-27 05:04:06 on dist-test-slave-39f2"
server_key: "acaf68dbdc82d0acdce2ff2720599377"
server_key_iv: "581e6bc6cbc2ec893958887b66f7fe31"
server_key_version: "encryptionkey@0"
I20260927 05:04:06.470805 8716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:06.482646 8716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:06.483420 8716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:06.483662 8716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:06.502507 8716 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:39223
I20260927 05:04:06.502526 8781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:39223 every 8 connection(s)
I20260927 05:04:06.503876 8716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:06.505681 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8716
I20260927 05:04:06.505884 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:06.506448 21157 external_mini_cluster.cc:1428] Setting key 868542f1f6a8fa86f6c8d50d0a73b95d
I20260927 05:04:06.508524 8782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:06.519695 8785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:06.508393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.20.169.126:39223)
0927 05:04:06.509392 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:06.509402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:06.509442 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:04:06.511090 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:06.511111 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:06.511547 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:06.511951 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:06.512511 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:06.513875 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:06.515230 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:06.515876 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:06.516081 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:06.517313 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:06.517321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:06.517336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:06.517886 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:06.517968 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:06.517993 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:06.518138 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:06.518353 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 05:04:06.518637 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":390,"threads_started":1}
I20260927 05:04:06.519953 8782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31: Bootstrap starting.
I20260927 05:04:06.523355 8782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:06.524264 8782 log.cc:824] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:06.526938 8782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31: No bootstrap required, opened a new log
I20260927 05:04:06.534107 8782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39223 } }
I20260927 05:04:06.534544 8782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:06.534679 8782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b56a79938704b41b2c3a6abf5fd9e31, State: Initialized, Role: FOLLOWER
I20260927 05:04:06.535362 8782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39223 } }
I20260927 05:04:06.535627 8782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:06.535763 8782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:06.535944 8782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:06.538247 8782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39223 } }
I20260927 05:04:06.538777 8782 leader_election.cc:302] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b56a79938704b41b2c3a6abf5fd9e31; no voters:
I20260927 05:04:06.539448 8782 leader_election.cc:288] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:06.539660 8787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:06.540705 8787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [term 1 LEADER]: Becoming Leader. State: Replica: 6b56a79938704b41b2c3a6abf5fd9e31, State: Running, Role: LEADER
I20260927 05:04:06.541225 8787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39223 } }
I20260927 05:04:06.541671 8782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:06.544137 8789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b56a79938704b41b2c3a6abf5fd9e31. Latest consensus state: current_term: 1 leader_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39223 } } }
I20260927 05:04:06.544647 8789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:06.544981 8788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b56a79938704b41b2c3a6abf5fd9e31" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 39223 } } }
I20260927 05:04:06.548421 8788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:06.551173 8796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:06.556365 8796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:06.565764 8796 catalog_manager.cc:1347] Generated new cluster ID: 5d3ebc29dd1d47d3afd340becf4583a8
I20260927 05:04:06.565923 8796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:06.580271 8796 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:06.582093 8796 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:06.590840 8796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31: Generated new TSK 0
I20260927 05:04:06.591751 8796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:06.595919 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.65:0
--local_ip_for_outbound_sockets=127.20.169.65
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39223
--builtin_ntp_servers=127.20.169.84:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:04:06.937261 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:06.937570 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:06.937733 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:06.945216 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:06.945348 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:06.945520 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.65
I20260927 05:04:06.954744 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.169.65
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.65
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:06.956012 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:06.958160 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:06.971225 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:06.971299 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:06.971276 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:06.972151 8807 server_base.cc:1034] running on GCE node
I20260927 05:04:06.972755 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:06.973754 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:06.974956 8807 hybrid_clock.cc:648] HybridClock initialized: now 1790485446974889 us; error 47 us; skew 500 ppm
I20260927 05:04:06.975327 8807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:04:06.980271 8807 webserver.cc:460] Webserver started at http://127.20.169.65:42769/ using document root <none> and password file <none>
I20260927 05:04:06.981024 8807 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:06.981117 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:06.981367 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:06.984174 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data/instance:
uuid: "b8b50020af814e15bde7297287ea4a05"
format_stamp: "Formatted at 2026-09-27 05:04:06 on dist-test-slave-39f2"
server_key: "0ea84aeafbc848c55307f95a930009cc"
server_key_iv: "980ce33a94c7291857292c227079c916"
server_key_version: "encryptionkey@0"
I20260927 05:04:06.984974 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance:
uuid: "b8b50020af814e15bde7297287ea4a05"
format_stamp: "Formatted at 2026-09-27 05:04:06 on dist-test-slave-39f2"
server_key: "0ea84aeafbc848c55307f95a930009cc"
server_key_iv: "980ce33a94c7291857292c227079c916"
server_key_version: "encryptionkey@0"
I20260927 05:04:06.990398 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:04:06.993865 8821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:06.995213 8807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:06.995404 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal
uuid: "b8b50020af814e15bde7297287ea4a05"
format_stamp: "Formatted at 2026-09-27 05:04:06 on dist-test-slave-39f2"
server_key: "0ea84aeafbc848c55307f95a930009cc"
server_key_iv: "980ce33a94c7291857292c227079c916"
server_key_version: "encryptionkey@0"
I20260927 05:04:06.995641 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:07.007885 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:07.008769 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:07.009115 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:07.010128 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:07.011639 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:07.011758 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.011897 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:07.011986 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.045974 8807 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.65:32807
I20260927 05:04:07.046015 8934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.65:32807 every 8 connection(s)
I20260927 05:04:07.047405 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/data/info.pb
I20260927 05:04:07.057166 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8807
I20260927 05:04:07.057361 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal/instance
I20260927 05:04:07.057902 21157 external_mini_cluster.cc:1428] Setting key 248260c0d1e262ef792dd370b92a23e6
I20260927 05:04:07.060794 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.66:0
--local_ip_for_outbound_sockets=127.20.169.66
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39223
--builtin_ntp_servers=127.20.169.84:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:04:07.060894 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:07.050218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.65:37321 (local address 127.20.169.126:39223)
0927 05:04:07.050710 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:07.050714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:07.052018 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 05:04:07.053898 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:07.053903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:07.053996 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:07.054119 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:07.055238 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.055987 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.058076 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.058488 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.058637 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:07.060075 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:07.060080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:07.060083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:07.060099 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:07.060136 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:07.060147 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:07.060240 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:07.060454 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 05:04:07.060546 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":241,"threads_started":1}
I20260927 05:04:07.061843 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:07.050811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39223 (local address 127.20.169.65:37321)
0927 05:04:07.051829 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:07.051867 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:07.053614 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:07.054364 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:07.054372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:07.054460 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:07.055053 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:07.055071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.056160 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.056164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:07.057909 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:07.057926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.058720 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.058729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:07.059015 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:07.059754 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:04:07.059758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:07.059809 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:04:07.059882 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:07.060239 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:07.060252 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:07.060262 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:04:07.060450 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 05:04:07.060848 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":285,"threads_started":1}
I20260927 05:04:07.063300 8935 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39223
I20260927 05:04:07.063623 8935 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:07.064543 8935 heartbeater.cc:511] Master 127.20.169.126:39223 requested a full tablet report, sending...
I20260927 05:04:07.066743 8747 ts_manager.cc:194] Registered new tserver with Master: b8b50020af814e15bde7297287ea4a05 (127.20.169.65:32807)
I20260927 05:04:07.068205 8747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.65:37321
W20260927 05:04:07.387859 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:07.388127 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:07.388267 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:07.395452 8939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:07.395550 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:07.395726 8939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.66
I20260927 05:04:07.404801 8939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.169.66
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.66
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:07.406062 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:07.407804 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:07.420512 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:07.420476 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:07.420548 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:07.421797 8939 server_base.cc:1034] running on GCE node
I20260927 05:04:07.422437 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:07.423341 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:07.424562 8939 hybrid_clock.cc:648] HybridClock initialized: now 1790485447424518 us; error 41 us; skew 500 ppm
I20260927 05:04:07.424944 8939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:04:07.427268 8939 webserver.cc:460] Webserver started at http://127.20.169.66:34009/ using document root <none> and password file <none>
I20260927 05:04:07.427939 8939 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:07.428072 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:07.428385 8939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:07.431241 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data/instance:
uuid: "245e656179c242dfb0528230214ec0fa"
format_stamp: "Formatted at 2026-09-27 05:04:07 on dist-test-slave-39f2"
server_key: "85a0797f06bcbf1cbdbead3b8b99179a"
server_key_iv: "6b83e6557097a72720c29159f323e51e"
server_key_version: "encryptionkey@0"
I20260927 05:04:07.432020 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance:
uuid: "245e656179c242dfb0528230214ec0fa"
format_stamp: "Formatted at 2026-09-27 05:04:07 on dist-test-slave-39f2"
server_key: "85a0797f06bcbf1cbdbead3b8b99179a"
server_key_iv: "6b83e6557097a72720c29159f323e51e"
server_key_version: "encryptionkey@0"
I20260927 05:04:07.437053 8939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:04:07.440339 8953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.441599 8939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:04:07.441774 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal
uuid: "245e656179c242dfb0528230214ec0fa"
format_stamp: "Formatted at 2026-09-27 05:04:07 on dist-test-slave-39f2"
server_key: "85a0797f06bcbf1cbdbead3b8b99179a"
server_key_iv: "6b83e6557097a72720c29159f323e51e"
server_key_version: "encryptionkey@0"
I20260927 05:04:07.441975 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:07.471225 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:07.472103 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:07.472409 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:07.473516 8939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:07.475178 8939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:07.475301 8939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.475461 8939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:07.475551 8939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.518811 8939 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.66:42937
I20260927 05:04:07.518885 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.66:42937 every 8 connection(s)
I20260927 05:04:07.520105 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/data/info.pb
I20260927 05:04:07.528591 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 8939
I20260927 05:04:07.528787 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal/instance
I20260927 05:04:07.529392 21157 external_mini_cluster.cc:1428] Setting key af8a53552c96953697948711a1b33db0
I20260927 05:04:07.532574 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.169.67:0
--local_ip_for_outbound_sockets=127.20.169.67
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39223
--builtin_ntp_servers=127.20.169.84:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:04:07.533375 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:07.522715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.66:40741 (local address 127.20.169.126:39223)
0927 05:04:07.522934 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:07.522940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:07.524420 (+ 1480us) server_negotiation.cc:407] Connection header received
0927 05:04:07.526259 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:07.526263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:07.526328 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:07.526415 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:07.527567 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.528261 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.530411 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.530830 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.530925 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:07.532376 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:07.532381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:07.532384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:07.532400 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:07.532429 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:07.532441 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:07.532509 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:07.532857 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 05:04:07.532967 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:04:07.535463 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:07.523349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39223 (local address 127.20.169.66:40741)
0927 05:04:07.524250 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:07.524294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:07.526003 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:07.526726 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:07.526735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:07.526852 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:07.527417 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:07.527434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.528416 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.528422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:07.530274 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:07.530286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:07.531053 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:07.531061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:07.531309 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:07.532030 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:04:07.532034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:07.532086 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:04:07.532184 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:07.532545 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:07.532558 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:07.532569 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:04:07.533591 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0927 05:04:07.534034 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":271,"threads_started":1}
I20260927 05:04:07.536803 9067 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39223
I20260927 05:04:07.537132 9067 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:07.537860 9067 heartbeater.cc:511] Master 127.20.169.126:39223 requested a full tablet report, sending...
I20260927 05:04:07.539283 8747 ts_manager.cc:194] Registered new tserver with Master: 245e656179c242dfb0528230214ec0fa (127.20.169.66:42937)
I20260927 05:04:07.540509 8747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.66:40741
W20260927 05:04:07.861546 9070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:07.861805 9070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:07.861888 9070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:07.870946 9070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:07.871088 9070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:07.871280 9070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.169.67
I20260927 05:04:07.882118 9070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.169.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.169.67
--webserver_port=0
--tserver_master_addrs=127.20.169.126:39223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.169.67
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:07.883440 9070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:07.885334 9070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:07.898285 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:07.898320 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:07.898340 9078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:07.904322 9070 server_base.cc:1034] running on GCE node
I20260927 05:04:07.905028 9070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:07.905990 9070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:07.907227 9070 hybrid_clock.cc:648] HybridClock initialized: now 1790485447907169 us; error 52 us; skew 500 ppm
I20260927 05:04:07.907568 9070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:04:07.909848 9070 webserver.cc:460] Webserver started at http://127.20.169.67:45039/ using document root <none> and password file <none>
I20260927 05:04:07.910586 9070 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:07.910724 9070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:07.911038 9070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:07.914008 9070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data/instance:
uuid: "aad152f6d63643fe9d9305f75f8ef6b0"
format_stamp: "Formatted at 2026-09-27 05:04:07 on dist-test-slave-39f2"
server_key: "03e2e257ef7c9305f4433c1856c96b97"
server_key_iv: "ca318bc1fb5465cbaff3d3dcdd97f16c"
server_key_version: "encryptionkey@0"
I20260927 05:04:07.914799 9070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance:
uuid: "aad152f6d63643fe9d9305f75f8ef6b0"
format_stamp: "Formatted at 2026-09-27 05:04:07 on dist-test-slave-39f2"
server_key: "03e2e257ef7c9305f4433c1856c96b97"
server_key_iv: "ca318bc1fb5465cbaff3d3dcdd97f16c"
server_key_version: "encryptionkey@0"
I20260927 05:04:07.919958 9070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:04:07.923473 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.924963 9070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:04:07.925098 9070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal
uuid: "aad152f6d63643fe9d9305f75f8ef6b0"
format_stamp: "Formatted at 2026-09-27 05:04:07 on dist-test-slave-39f2"
server_key: "03e2e257ef7c9305f4433c1856c96b97"
server_key_iv: "ca318bc1fb5465cbaff3d3dcdd97f16c"
server_key_version: "encryptionkey@0"
I20260927 05:04:07.925273 9070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:07.956390 9070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:07.957255 9070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:07.957510 9070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:07.958482 9070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:04:07.959872 9070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:04:07.959955 9070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.960028 9070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:04:07.960083 9070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:07.994442 9070 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.67:32833
I20260927 05:04:07.994536 9197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.67:32833 every 8 connection(s)
I20260927 05:04:07.995754 9070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/data/info.pb
I20260927 05:04:08.001191 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9070
I20260927 05:04:08.001395 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal/instance
I20260927 05:04:08.001806 21157 external_mini_cluster.cc:1428] Setting key 29c8c87dc556b92fde6916327ce341bd
I20260927 05:04:08.009100 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:07.998549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:49869 (local address 127.20.169.126:39223)
0927 05:04:07.998754 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:07.998760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.000292 (+ 1532us) server_negotiation.cc:407] Connection header received
0927 05:04:08.002368 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.002374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.002462 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.002591 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:08.003712 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.004595 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.006324 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.006812 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.006931 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.008269 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:08.008275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:08.008278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:08.008298 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:08.008339 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:08.008354 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:08.008438 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.008657 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.008751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:04:08.010213 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:07.999076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:39223 (local address 127.20.169.67:49869)
0927 05:04:08.000067 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.000142 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:08.002090 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:08.002835 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:08.002843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:08.002936 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:08.003533 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.003555 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.004789 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.004795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.006153 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.006166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.007025 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.007030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.007246 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.007936 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:04:08.007938 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:08.007979 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:04:08.008059 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:08.008453 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:08.008461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:08.008468 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:04:08.008621 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 05:04:08.009057 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":345,"threads_started":1}
I20260927 05:04:08.011682 9198 heartbeater.cc:347] Connected to a master server at 127.20.169.126:39223
I20260927 05:04:08.012015 9198 heartbeater.cc:464] Registering TS with master...
I20260927 05:04:08.012786 9198 heartbeater.cc:511] Master 127.20.169.126:39223 requested a full tablet report, sending...
I20260927 05:04:08.014170 8747 ts_manager.cc:194] Registered new tserver with Master: aad152f6d63643fe9d9305f75f8ef6b0 (127.20.169.67:32833)
I20260927 05:04:08.015024 8747 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.169.67:49869
I20260927 05:04:08.017217 21157 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:04:08.027017 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.021349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.20.169.126:39223)
0927 05:04:08.021532 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.021537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.022062 (+ 525us) server_negotiation.cc:407] Connection header received
0927 05:04:08.022167 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.022171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.022245 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.022331 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:08.022803 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.023497 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.024912 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.025291 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.025398 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.026441 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:08.026446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:08.026449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:08.026466 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:08.026497 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:08.026507 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:08.026577 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.026714 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.026783 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:04:08.031438 8747 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:04:08.038864 8747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:04:08.061209 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.052309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:42937 (local address 127.0.0.1:38152)
0927 05:04:08.053824 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.053859 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:08.054041 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:08.054670 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:08.054677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:08.054722 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:08.055172 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.055188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.056650 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.056657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.058567 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.058582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.060348 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.060352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.060484 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.060519 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:04:08.060662 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 05:04:08.060867 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"thread_start_us":2067,"threads_started":1}
I20260927 05:04:08.061249 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.052529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.20.169.66:42937)
0927 05:04:08.053305 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.053317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.053990 (+ 673us) server_negotiation.cc:407] Connection header received
0927 05:04:08.054261 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.054267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.054457 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.054661 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:08.055365 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.056509 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.058764 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.059944 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.060027 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.060193 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.060714 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.060907 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":412,"threads_started":1}
I20260927 05:04:08.064517 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.052923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.67:32833 (local address 127.0.0.1:53516)
0927 05:04:08.054666 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.054682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:08.055065 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:08.055927 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:08.055934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:08.055966 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:08.056378 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.056392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.059070 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.059079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.061772 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.061786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.063731 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.063738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.063939 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.063975 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:04:08.064075 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:04:08.064208 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"thread_start_us":560,"threads_started":1}
I20260927 05:04:08.064780 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.053256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.20.169.67:32833)
0927 05:04:08.054012 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.054022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.054948 (+ 926us) server_negotiation.cc:407] Connection header received
0927 05:04:08.055299 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.055309 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.055568 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.055758 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:08.056837 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.058870 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.061990 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.063473 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.063682 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.063976 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.064131 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.064287 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":420,"threads_started":1}
I20260927 05:04:08.065896 9001 tablet_service.cc:1467] Processing CreateTablet for tablet 1f305341b1f54090b7bf121018d96e8d (DEFAULT_TABLE table=test-table [id=b44638ef8dc844879266e04f383d6cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:08.067298 9001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f305341b1f54090b7bf121018d96e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:08.067840 9132 tablet_service.cc:1467] Processing CreateTablet for tablet 1f305341b1f54090b7bf121018d96e8d (DEFAULT_TABLE table=test-table [id=b44638ef8dc844879266e04f383d6cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:08.069228 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f305341b1f54090b7bf121018d96e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:08.071146 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.054727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43874)
0927 05:04:08.057127 (+ 2400us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.057144 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:08.057255 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:08.063218 (+ 5963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:08.063222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:08.063246 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:08.063615 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.063626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.065516 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.065521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.068378 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.068393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.070374 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.070381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.070556 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.070589 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:04:08.070685 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:04:08.070826 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2267,"thread_start_us":2114,"threads_started":1}
I20260927 05:04:08.071440 8935 heartbeater.cc:503] Master 127.20.169.126:39223 was elected leader, sending a full tablet report...
I20260927 05:04:08.071930 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.055264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.20.169.65:32807)
0927 05:04:08.062389 (+ 7125us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.062395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.062417 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:04:08.062495 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.062500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.062746 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.062983 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:08.063825 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.065363 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.068588 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.070157 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.070315 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.070578 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.070755 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.070919 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6958,"thread_start_us":6846,"threads_started":1}
I20260927 05:04:08.075680 8869 tablet_service.cc:1467] Processing CreateTablet for tablet 1f305341b1f54090b7bf121018d96e8d (DEFAULT_TABLE table=test-table [id=b44638ef8dc844879266e04f383d6cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:04:08.077126 8869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f305341b1f54090b7bf121018d96e8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:08.081498 9216 tablet_bootstrap.cc:492] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Bootstrap starting.
I20260927 05:04:08.084858 9216 tablet_bootstrap.cc:654] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:08.086048 9216 log.cc:824] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:08.088987 9217 tablet_bootstrap.cc:492] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Bootstrap starting.
I20260927 05:04:08.089175 9216 tablet_bootstrap.cc:492] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: No bootstrap required, opened a new log
I20260927 05:04:08.089466 9216 ts_tablet_manager.cc:1397] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260927 05:04:08.092734 9217 tablet_bootstrap.cc:654] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:08.092948 9219 tablet_bootstrap.cc:492] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Bootstrap starting.
I20260927 05:04:08.094141 9217 log.cc:824] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:08.096401 9219 tablet_bootstrap.cc:654] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:08.097534 9219 log.cc:824] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:08.097610 9217 tablet_bootstrap.cc:492] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: No bootstrap required, opened a new log
I20260927 05:04:08.097899 9217 ts_tablet_manager.cc:1397] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260927 05:04:08.100090 9216 raft_consensus.cc:348] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.100567 9216 raft_consensus.cc:374] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:08.100605 9219 tablet_bootstrap.cc:492] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: No bootstrap required, opened a new log
I20260927 05:04:08.100674 9216 raft_consensus.cc:729] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad152f6d63643fe9d9305f75f8ef6b0, State: Initialized, Role: FOLLOWER
I20260927 05:04:08.100881 9219 ts_tablet_manager.cc:1397] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260927 05:04:08.101430 9216 consensus_queue.cc:260] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.102602 9198 heartbeater.cc:503] Master 127.20.169.126:39223 was elected leader, sending a full tablet report...
I20260927 05:04:08.103431 9216 ts_tablet_manager.cc:1428] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 05:04:08.109513 9219 raft_consensus.cc:348] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.109752 9217 raft_consensus.cc:348] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.110055 9219 raft_consensus.cc:374] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:08.110174 9219 raft_consensus.cc:729] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b50020af814e15bde7297287ea4a05, State: Initialized, Role: FOLLOWER
I20260927 05:04:08.110190 9217 raft_consensus.cc:374] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:08.110314 9217 raft_consensus.cc:729] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245e656179c242dfb0528230214ec0fa, State: Initialized, Role: FOLLOWER
I20260927 05:04:08.110807 9219 consensus_queue.cc:260] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.110872 9217 consensus_queue.cc:260] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.111936 9067 heartbeater.cc:503] Master 127.20.169.126:39223 was elected leader, sending a full tablet report...
I20260927 05:04:08.112654 9217 ts_tablet_manager.cc:1428] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 05:04:08.112649 9219 ts_tablet_manager.cc:1428] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260927 05:04:08.247529 9199 tablet.cc:2369] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:08.255731 9222 raft_consensus.cc:482] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:04:08.256060 9222 raft_consensus.cc:504] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.257828 9222 leader_election.cc:288] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 245e656179c242dfb0528230214ec0fa (127.20.169.66:42937), b8b50020af814e15bde7297287ea4a05 (127.20.169.65:32807)
I20260927 05:04:08.265219 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.258457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:48391 (local address 127.20.169.66:42937)
0927 05:04:08.258865 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.258874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.258900 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:04:08.258995 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.259003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.259122 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.259237 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:08.259937 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.261014 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.263142 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.264317 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.264426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.264474 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.264932 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.265029 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260927 05:04:08.265419 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.258225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.66:42937 (local address 127.20.169.67:48391)
0927 05:04:08.258396 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.258415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:08.258550 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:08.259476 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:08.259499 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:08.259537 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:08.259749 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.259759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.261170 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.261176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.262976 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.262990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.264473 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.264480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.264694 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.264740 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 05:04:08.264895 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:04:08.265046 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":22}
I20260927 05:04:08.266381 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.259046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.169.67:36173 (local address 127.20.169.65:32807)
0927 05:04:08.259377 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.259387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.259981 (+ 594us) server_negotiation.cc:407] Connection header received
0927 05:04:08.260135 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.260139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.260213 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.260279 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:04:08.260947 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.262050 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.264294 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.265631 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.265842 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.265926 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.266087 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.266193 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 05:04:08.266570 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.258667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.20.169.67:36173)
0927 05:04:08.259822 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.259861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:08.260014 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:08.260329 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:08.260340 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:08.260364 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:04:08.260737 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.260748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.262220 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.262224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.264061 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:08.264077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.265809 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.265814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:08.265934 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.265955 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 05:04:08.266027 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 05:04:08.266166 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":1,"thread_start_us":849,"threads_started":1}
I20260927 05:04:08.266628 9021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f305341b1f54090b7bf121018d96e8d" candidate_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245e656179c242dfb0528230214ec0fa" is_pre_election: true
I20260927 05:04:08.267210 9021 raft_consensus.cc:2454] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad152f6d63643fe9d9305f75f8ef6b0 in term 0.
I20260927 05:04:08.267666 8890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f305341b1f54090b7bf121018d96e8d" candidate_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b50020af814e15bde7297287ea4a05" is_pre_election: true
I20260927 05:04:08.268003 9088 leader_election.cc:302] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245e656179c242dfb0528230214ec0fa, aad152f6d63643fe9d9305f75f8ef6b0; no voters:
I20260927 05:04:08.268131 8890 raft_consensus.cc:2454] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad152f6d63643fe9d9305f75f8ef6b0 in term 0.
I20260927 05:04:08.268405 9222 raft_consensus.cc:2781] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:04:08.268555 9222 raft_consensus.cc:482] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:04:08.268694 9222 raft_consensus.cc:3037] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:08.270874 9222 raft_consensus.cc:504] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.271435 9222 leader_election.cc:288] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [CANDIDATE]: Term 1 election: Requested vote from peers 245e656179c242dfb0528230214ec0fa (127.20.169.66:42937), b8b50020af814e15bde7297287ea4a05 (127.20.169.65:32807)
I20260927 05:04:08.271915 9021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f305341b1f54090b7bf121018d96e8d" candidate_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245e656179c242dfb0528230214ec0fa"
I20260927 05:04:08.272065 8890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f305341b1f54090b7bf121018d96e8d" candidate_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b50020af814e15bde7297287ea4a05"
W20260927 05:04:08.272136 9068 tablet.cc:2369] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:08.272133 9021 raft_consensus.cc:3037] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:08.272312 8890 raft_consensus.cc:3037] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:08.274756 8890 raft_consensus.cc:2454] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad152f6d63643fe9d9305f75f8ef6b0 in term 1.
I20260927 05:04:08.274785 9021 raft_consensus.cc:2454] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad152f6d63643fe9d9305f75f8ef6b0 in term 1.
I20260927 05:04:08.275120 9088 leader_election.cc:302] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad152f6d63643fe9d9305f75f8ef6b0, b8b50020af814e15bde7297287ea4a05; no voters:
I20260927 05:04:08.275408 9222 raft_consensus.cc:2781] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:08.275876 9222 raft_consensus.cc:686] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 1 LEADER]: Becoming Leader. State: Replica: aad152f6d63643fe9d9305f75f8ef6b0, State: Running, Role: LEADER
I20260927 05:04:08.276419 9222 consensus_queue.cc:237] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } }
I20260927 05:04:08.280982 8747 catalog_manager.cc:5495] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad152f6d63643fe9d9305f75f8ef6b0 (127.20.169.67). New cstate: current_term: 1 leader_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad152f6d63643fe9d9305f75f8ef6b0" member_type: VOTER last_known_addr { host: "127.20.169.67" port: 32833 } health_report { overall_health: HEALTHY } } }
W20260927 05:04:08.299810 8936 tablet.cc:2369] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:08.302330 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.296693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.20.169.67:32833)
0927 05:04:08.296945 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.296952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.297044 (+ 92us) server_negotiation.cc:407] Connection header received
0927 05:04:08.297215 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.297218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.297288 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.297408 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:08.297722 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.298424 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.300007 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.300534 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.300675 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.301241 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:08.301739 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:08.301914 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.301977 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.302059 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:04:08.310436 8890 raft_consensus.cc:1261] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 1 FOLLOWER]: Refusing update from remote peer aad152f6d63643fe9d9305f75f8ef6b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:08.310437 9021 raft_consensus.cc:1261] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 1 FOLLOWER]: Refusing update from remote peer aad152f6d63643fe9d9305f75f8ef6b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:08.311331 9226 consensus_queue.cc:1035] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245e656179c242dfb0528230214ec0fa" member_type: VOTER last_known_addr { host: "127.20.169.66" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:08.311699 9222 consensus_queue.cc:1035] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8b50020af814e15bde7297287ea4a05" member_type: VOTER last_known_addr { host: "127.20.169.65" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:08.335947 8746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:04:08.336228 8746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:04:08.339715 8746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b56a79938704b41b2c3a6abf5fd9e31: Sending DeleteTablet for 3 replicas of tablet 1f305341b1f54090b7bf121018d96e8d
I20260927 05:04:08.340709 9001 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f305341b1f54090b7bf121018d96e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:08 UTC) from {username='slave'} at 127.0.0.1:38152
I20260927 05:04:08.340878 8869 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f305341b1f54090b7bf121018d96e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:08 UTC) from {username='slave'} at 127.0.0.1:43874
I20260927 05:04:08.340866 9132 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f305341b1f54090b7bf121018d96e8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:04:08 UTC) from {username='slave'} at 127.0.0.1:53516
I20260927 05:04:08.344151 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8807
I20260927 05:04:08.354188 9240 tablet_replica.cc:330] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: stopping tablet replica
I20260927 05:04:08.354780 9241 tablet_replica.cc:330] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: stopping tablet replica
I20260927 05:04:08.355230 9241 raft_consensus.cc:2229] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:04:08.355734 9241 raft_consensus.cc:2258] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:08.357204 9240 raft_consensus.cc:2229] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:04:08.357857 9240 raft_consensus.cc:2258] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:08.358850 9241 ts_tablet_manager.cc:1905] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:08.361007 9240 ts_tablet_manager.cc:1905] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:08.364771 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.345912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.20.169.65:32807)
0927 05:04:08.346140 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.346149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.356638 (+ 10489us) server_negotiation.cc:407] Connection header received
0927 05:04:08.356866 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.356871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.356970 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.357082 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:08.357843 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.358933 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.360871 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.361509 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.361599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.363059 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:08.363070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:08.363086 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:08.363839 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:08.363899 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:08.363926 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:08.364005 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.364258 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.364367 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:04:08.366169 8930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:08.366451 9239 tablet_replica.cc:330] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: stopping tablet replica
I20260927 05:04:08.366941 9239 raft_consensus.cc:2229] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:04:08.367705 9239 raft_consensus.cc:2258] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:04:08.369575 9240 ts_tablet_manager.cc:1918] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:08.369750 9240 log.cc:1196] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-2/wal/wals/1f305341b1f54090b7bf121018d96e8d
I20260927 05:04:08.370188 9239 ts_tablet_manager.cc:1905] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:04:08.370289 9240 ts_tablet_manager.cc:1939] T 1f305341b1f54090b7bf121018d96e8d P aad152f6d63643fe9d9305f75f8ef6b0: Deleting consensus metadata
I20260927 05:04:08.372051 8733 catalog_manager.cc:4841] TS aad152f6d63643fe9d9305f75f8ef6b0 (127.20.169.67:32833): tablet 1f305341b1f54090b7bf121018d96e8d (table test-table [id=b44638ef8dc844879266e04f383d6cba]) successfully deleted
I20260927 05:04:08.379789 9239 ts_tablet_manager.cc:1918] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:08.379976 9239 log.cc:1196] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-1/wal/wals/1f305341b1f54090b7bf121018d96e8d
I20260927 05:04:08.380422 9239 ts_tablet_manager.cc:1939] T 1f305341b1f54090b7bf121018d96e8d P 245e656179c242dfb0528230214ec0fa: Deleting consensus metadata
I20260927 05:04:08.381970 8734 catalog_manager.cc:4841] TS 245e656179c242dfb0528230214ec0fa (127.20.169.66:42937): tablet 1f305341b1f54090b7bf121018d96e8d (table test-table [id=b44638ef8dc844879266e04f383d6cba]) successfully deleted
I20260927 05:04:08.486238 9241 ts_tablet_manager.cc:1918] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:04:08.486537 9241 log.cc:1196] T 1f305341b1f54090b7bf121018d96e8d P b8b50020af814e15bde7297287ea4a05: Deleting WAL directory at /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485126081201-21157-0/minicluster-data/ts-0/wal/wals/1f305341b1f54090b7bf121018d96e8d
I20260927 05:04:08.486997 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8807
W20260927 05:04:08.505996 8734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.169.65:32807 (error 108)
I20260927 05:04:08.506718 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8939
I20260927 05:04:08.508955 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.508356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43896)
0927 05:04:08.508612 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.508743 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260927 05:04:08.509227 8734 catalog_manager.cc:4584] TS b8b50020af814e15bde7297287ea4a05 (127.20.169.65:32807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f305341b1f54090b7bf121018d96e8d: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
I20260927 05:04:08.518625 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.507881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.20.169.66:42937)
0927 05:04:08.508046 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.508054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.508074 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:04:08.508139 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.508144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.508274 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.508388 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:08.509236 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.510095 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.511448 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.511849 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.511990 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.517200 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:08.517210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:08.517225 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:08.517806 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:08.517854 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:08.517891 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:08.518015 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.518122 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.518234 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 05:04:08.519487 9062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:08.563114 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.562411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43904)
0927 05:04:08.562600 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.562709 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:04:08.564702 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.564015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43920)
0927 05:04:08.564478 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.564563 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 05:04:08.625895 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.625272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43926)
0927 05:04:08.625531 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.625653 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 05:04:08.627194 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.626761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43942)
0927 05:04:08.626948 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.627040 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 05:04:08.650265 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8939
I20260927 05:04:08.668664 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9070
I20260927 05:04:08.677289 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.669766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.20.169.67:32833)
0927 05:04:08.669959 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:08.669964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:08.669983 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:04:08.670059 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:08.670066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:08.670158 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:08.670257 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:08.670764 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.671673 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.673732 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:08.674229 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:08.674434 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:08.675729 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:08.675739 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:08.675753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:08.676368 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:08.676423 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:08.676444 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:08.676535 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:08.676713 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 05:04:08.676906 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:04:08.678145 9193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:08.707973 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.707451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43948)
0927 05:04:08.707676 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.707778 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 05:04:08.709185 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.708681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43962)
0927 05:04:08.708995 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.709077 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260927 05:04:08.797824 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9070
I20260927 05:04:08.818131 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 8716
I20260927 05:04:08.819325 8777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:08.929791 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:08.929209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.65:32807 (local address 127.0.0.1:43976)
0927 05:04:08.929464 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:08.929566 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.65:32807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 05:04:08.929821 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 8716
2026-09-27T05:04:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2936 ms)
[----------] 6 tests from EncryptionPolicyTest (22611 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:09 dist-test-slave-39f2 krb5kdc[9250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:09 dist-test-slave-39f2 krb5kdc[9250](info): set up 2 sockets
Sep 27 05:04:09 dist-test-slave-39f2 krb5kdc[9250](info): commencing operation
krb5kdc: starting...
W20260927 05:04:10.109097 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:10 dist-test-slave-39f2 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:10.367616 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:10.704164 9263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:10.704517 9263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:10.704675 9263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:10.711915 9263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:10.712057 9263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:10.712162 9263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:10.712261 9263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:10.712355 9263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:10.722397 9263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:10.723697 9263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:10.725023 9263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:10.737063 9268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:10.737113 9269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:10.737140 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:10.737963 9263 server_base.cc:1034] running on GCE node
I20260927 05:04:10.738751 9263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:10.740084 9263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:10.741333 9263 hybrid_clock.cc:648] HybridClock initialized: now 1790485450741261 us; error 70 us; skew 500 ppm
I20260927 05:04:10.741664 9263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:10 dist-test-slave-39f2 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:10.744681 9263 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:10.746116 9263 webserver.cc:460] Webserver started at http://127.0.0.1:37291/ using document root <none> and password file <none>
I20260927 05:04:10.746758 9263 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:10.746905 9263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:10.747192 9263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:10.750118 9263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "0fc1f9e638474a789b994378d6705f60"
format_stamp: "Formatted at 2026-09-27 05:04:10 on dist-test-slave-39f2"
server_key: "39eb22cfae2e7a642704038d112d6a94"
server_key_iv: "d41c82698a49aa584bf350291e14c294"
server_key_version: "encryptionkey@0"
I20260927 05:04:10.750883 9263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "0fc1f9e638474a789b994378d6705f60"
format_stamp: "Formatted at 2026-09-27 05:04:10 on dist-test-slave-39f2"
server_key: "39eb22cfae2e7a642704038d112d6a94"
server_key_iv: "d41c82698a49aa584bf350291e14c294"
server_key_version: "encryptionkey@0"
I20260927 05:04:10.755756 9263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:04:10.759063 9278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:10.760318 9263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:04:10.760496 9263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "0fc1f9e638474a789b994378d6705f60"
format_stamp: "Formatted at 2026-09-27 05:04:10 on dist-test-slave-39f2"
server_key: "39eb22cfae2e7a642704038d112d6a94"
server_key_iv: "d41c82698a49aa584bf350291e14c294"
server_key_version: "encryptionkey@0"
I20260927 05:04:10.760679 9263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:10.782512 9263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:10.786605 9263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:10.786882 9263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:10.805491 9263 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35411
I20260927 05:04:10.805568 9330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35411 every 8 connection(s)
I20260927 05:04:10.806780 9263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:10.810542 9331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:10.810703 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9263
I20260927 05:04:10.810926 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:10.811811 21157 external_mini_cluster.cc:1428] Setting key 13c108e58404504e0d2e29a73b0740be
I20260927 05:04:10.824851 9331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60: Bootstrap starting.
Sep 27 05:04:10 dist-test-slave-39f2 krb5kdc[9250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:04:10.828563 9331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:10.829869 9331 log.cc:824] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:10.832863 9331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60: No bootstrap required, opened a new log
I20260927 05:04:10.835608 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:10.813797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57569 (local address 127.0.0.1:35411)
0927 05:04:10.814582 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:10.814596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:10.814638 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:04:10.816793 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:10.816863 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:10.817276 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:10.817620 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:10.819380 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:10.820688 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:10.822560 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:10.823200 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:10.823367 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:10.828140 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:10.828148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:10.828163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:10.828199 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:10.831405 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:10.832231 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:10.832239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:10.832245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:10.832415 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:10.832919 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:10.832926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:10.832929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:10.833464 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:10.833805 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:10.834282 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 05:04:10.834603 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":369,"threads_started":1}
I20260927 05:04:10.839548 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:10.841192 9331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1f9e638474a789b994378d6705f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 05:04:10.841533 9331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:10.841655 9331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc1f9e638474a789b994378d6705f60, State: Initialized, Role: FOLLOWER
I20260927 05:04:10.842196 9331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1f9e638474a789b994378d6705f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 05:04:10.842444 9331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:10.842595 9331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:10.842784 9331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:10.845501 9331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1f9e638474a789b994378d6705f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 05:04:10.846124 9331 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc1f9e638474a789b994378d6705f60; no voters:
I20260927 05:04:10.846724 9331 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:10.846880 9341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:10.847551 9341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [term 1 LEADER]: Becoming Leader. State: Replica: 0fc1f9e638474a789b994378d6705f60, State: Running, Role: LEADER
I20260927 05:04:10.848131 9341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1f9e638474a789b994378d6705f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260927 05:04:10.851642 9341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc1f9e638474a789b994378d6705f60. Latest consensus state: current_term: 1 leader_uuid: "0fc1f9e638474a789b994378d6705f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1f9e638474a789b994378d6705f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } } }
I20260927 05:04:10.851918 9341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:10.853094 9343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc1f9e638474a789b994378d6705f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc1f9e638474a789b994378d6705f60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } } }
I20260927 05:04:10.853551 9343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:10.855041 9331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:10.861936 9345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:10.867549 9345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:10.867880 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:10.844423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56825 (local address 127.0.0.1:35411)
0927 05:04:10.844620 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:10.844627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:10.845364 (+ 737us) server_negotiation.cc:407] Connection header received
0927 05:04:10.845495 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:10.845501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:10.845580 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:10.845694 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:10.847556 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:10.848712 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:10.851155 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:10.851867 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:10.854608 (+ 2741us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:10.857964 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:10.857973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:10.857981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:10.858024 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:10.863399 (+ 5375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:10.864482 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:10.864490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:10.864496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:10.864678 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:10.865314 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:10.865320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:10.865323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:10.865751 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:10.866028 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:10.867284 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0927 05:04:10.867477 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:04:10.877051 9345 catalog_manager.cc:1347] Generated new cluster ID: dc1a9974139e4503865ea90436568703
I20260927 05:04:10.877172 9345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:10.889468 9345 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:10.890671 9345 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:10.899055 9345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc1f9e638474a789b994378d6705f60: Generated new TSK 0
I20260927 05:04:10.899861 9345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:10.906628 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9263
I20260927 05:04:10.907982 9326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:11.004271 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9263
W20260927 05:04:11.022564 9248 connection.cc:535] client connection to 127.0.0.1:35411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35411): Connection reset by peer (error 104)
2026-09-27T05:04:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:11 dist-test-slave-39f2 krb5kdc[9363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:11 dist-test-slave-39f2 krb5kdc[9363](info): set up 2 sockets
Sep 27 05:04:11 dist-test-slave-39f2 krb5kdc[9363](info): commencing operation
krb5kdc: starting...
W20260927 05:04:12.164710 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:12 dist-test-slave-39f2 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:12.423995 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:12.759172 9376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:12.759495 9376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:12.759670 9376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:12.766638 9376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:12.766798 9376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:12.766899 9376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:12.767011 9376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:12.767095 9376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:12.776649 9376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42175
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:12.777915 9376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:12.779211 9376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:12.790735 9382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:12.790773 9384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:12.790840 9381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:12.791646 9376 server_base.cc:1034] running on GCE node
I20260927 05:04:12.792315 9376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:12.793730 9376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:12.794940 9376 hybrid_clock.cc:648] HybridClock initialized: now 1790485452794879 us; error 50 us; skew 500 ppm
I20260927 05:04:12.795234 9376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:12 dist-test-slave-39f2 krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:12.802692 9376 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:12.804026 9376 webserver.cc:460] Webserver started at http://127.0.0.1:42229/ using document root <none> and password file <none>
I20260927 05:04:12.804648 9376 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:12.804749 9376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:12.805058 9376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:12.807893 9376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "1e3df9e1476a4328bf8faf5f76503065"
format_stamp: "Formatted at 2026-09-27 05:04:12 on dist-test-slave-39f2"
server_key: "66feb4227225f169c3a4379aaaee2351"
server_key_iv: "4def5302b2679736777b669c9157aef7"
server_key_version: "encryptionkey@0"
I20260927 05:04:12.808597 9376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "1e3df9e1476a4328bf8faf5f76503065"
format_stamp: "Formatted at 2026-09-27 05:04:12 on dist-test-slave-39f2"
server_key: "66feb4227225f169c3a4379aaaee2351"
server_key_iv: "4def5302b2679736777b669c9157aef7"
server_key_version: "encryptionkey@0"
I20260927 05:04:12.813966 9376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:04:12.817725 9391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:12.819164 9376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 05:04:12.819442 9376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "1e3df9e1476a4328bf8faf5f76503065"
format_stamp: "Formatted at 2026-09-27 05:04:12 on dist-test-slave-39f2"
server_key: "66feb4227225f169c3a4379aaaee2351"
server_key_iv: "4def5302b2679736777b669c9157aef7"
server_key_version: "encryptionkey@0"
I20260927 05:04:12.819628 9376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:12.840921 9376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:12.844916 9376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:12.845197 9376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:12.864522 9376 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42175
I20260927 05:04:12.864609 9442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42175 every 8 connection(s)
I20260927 05:04:12.865934 9376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:12.869901 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9376
I20260927 05:04:12.870141 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:12.870303 9443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:12.870718 21157 external_mini_cluster.cc:1428] Setting key 4cd49e08580fdb43e98e1db080c4097b
I20260927 05:04:12.884207 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065: Bootstrap starting.
I20260927 05:04:12.884402 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:12.872655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54999 (local address 127.0.0.1:42175)
0927 05:04:12.873563 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:12.873577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:12.873617 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:04:12.875589 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:12.875611 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:12.876035 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:12.876456 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:12.877048 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:12.878409 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:12.879821 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:12.880501 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:12.880720 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:12.882135 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:12.882143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:12.882157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:12.882675 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:12.882751 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:12.882776 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:12.882973 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:12.883266 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 05:04:12.883433 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":364,"threads_started":1}
I20260927 05:04:12.887791 9443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:12.888173 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:12.889029 9443 log.cc:824] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:12.892045 9443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065: No bootstrap required, opened a new log
I20260927 05:04:12.898864 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:12.892583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60431 (local address 127.0.0.1:42175)
0927 05:04:12.892737 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:12.892741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:12.893304 (+ 563us) server_negotiation.cc:407] Connection header received
0927 05:04:12.893534 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:12.893539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:12.893596 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:12.893695 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:12.894197 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:12.894985 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:12.896316 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:12.896801 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:12.897011 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:12.898161 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:12.898166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:12.898169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:12.898185 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:12.898220 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:12.898237 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:12.898379 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:12.898565 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 05:04:12.898616 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:04:12.901347 9443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3df9e1476a4328bf8faf5f76503065" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260927 05:04:12.901742 9443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:12.901861 9443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3df9e1476a4328bf8faf5f76503065, State: Initialized, Role: FOLLOWER
I20260927 05:04:12.902421 9443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3df9e1476a4328bf8faf5f76503065" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260927 05:04:12.902683 9443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:12.902832 9443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:12.903014 9443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:12.905701 9443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3df9e1476a4328bf8faf5f76503065" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260927 05:04:12.906234 9443 leader_election.cc:302] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e3df9e1476a4328bf8faf5f76503065; no voters:
I20260927 05:04:12.906857 9443 leader_election.cc:288] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:12.907002 9453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:12.907737 9453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [term 1 LEADER]: Becoming Leader. State: Replica: 1e3df9e1476a4328bf8faf5f76503065, State: Running, Role: LEADER
I20260927 05:04:12.908267 9453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3df9e1476a4328bf8faf5f76503065" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260927 05:04:12.909097 9443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:12.911451 9454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e3df9e1476a4328bf8faf5f76503065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3df9e1476a4328bf8faf5f76503065" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } } }
I20260927 05:04:12.911855 9454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:12.911755 9455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e3df9e1476a4328bf8faf5f76503065. Latest consensus state: current_term: 1 leader_uuid: "1e3df9e1476a4328bf8faf5f76503065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3df9e1476a4328bf8faf5f76503065" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } } }
I20260927 05:04:12.912061 9455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:12.914363 9459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:12.920542 9459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:12.930420 9459 catalog_manager.cc:1347] Generated new cluster ID: 39660f1e0acd49d8935846afc016c2d8
I20260927 05:04:12.930532 9459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:12.962826 9459 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:12.964339 9459 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:12.976438 9459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e3df9e1476a4328bf8faf5f76503065: Generated new TSK 0
I20260927 05:04:12.977375 9459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:12.989208 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9376
W20260927 05:04:12.990396 9438 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54999
W20260927 05:04:12.991194 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:12.991456 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9376
W20260927 05:04:13.010497 9361 connection.cc:535] client connection to 127.0.0.1:42175 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42175): Connection reset by peer (error 104)
2026-09-27T05:04:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:13 dist-test-slave-39f2 krb5kdc[9474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:13 dist-test-slave-39f2 krb5kdc[9474](info): set up 2 sockets
Sep 27 05:04:13 dist-test-slave-39f2 krb5kdc[9474](info): commencing operation
krb5kdc: starting...
W20260927 05:04:14.105105 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:14 dist-test-slave-39f2 krb5kdc[9474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:14.365096 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:14.694687 9487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:14.695031 9487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:14.695215 9487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:14.702373 9487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:14.702539 9487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:14.702630 9487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:14.702723 9487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:14.702817 9487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:14.712649 9487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:14.714013 9487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:14.715422 9487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:14.726867 9492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:14.726955 9495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:14.726966 9493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:14.727689 9487 server_base.cc:1034] running on GCE node
I20260927 05:04:14.728463 9487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:14.730078 9487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:14.731329 9487 hybrid_clock.cc:648] HybridClock initialized: now 1790485454731273 us; error 49 us; skew 500 ppm
I20260927 05:04:14.731698 9487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:14 dist-test-slave-39f2 krb5kdc[9474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:14.734810 9487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:14.736248 9487 webserver.cc:460] Webserver started at http://127.0.0.1:37507/ using document root <none> and password file <none>
I20260927 05:04:14.737015 9487 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:14.737162 9487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:14.737461 9487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:14.740396 9487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "d16eb8e55eb24cfbadc3e3857c226bf5"
format_stamp: "Formatted at 2026-09-27 05:04:14 on dist-test-slave-39f2"
server_key: "5bf5429cda10bd6ba99926f2e10f3dc1"
server_key_iv: "d592e0267ce71b6f6902acd6ec69ecce"
server_key_version: "encryptionkey@0"
I20260927 05:04:14.741212 9487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "d16eb8e55eb24cfbadc3e3857c226bf5"
format_stamp: "Formatted at 2026-09-27 05:04:14 on dist-test-slave-39f2"
server_key: "5bf5429cda10bd6ba99926f2e10f3dc1"
server_key_iv: "d592e0267ce71b6f6902acd6ec69ecce"
server_key_version: "encryptionkey@0"
I20260927 05:04:14.746340 9487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 05:04:14.749658 9502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:14.751155 9487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:04:14.751349 9487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "d16eb8e55eb24cfbadc3e3857c226bf5"
format_stamp: "Formatted at 2026-09-27 05:04:14 on dist-test-slave-39f2"
server_key: "5bf5429cda10bd6ba99926f2e10f3dc1"
server_key_iv: "d592e0267ce71b6f6902acd6ec69ecce"
server_key_version: "encryptionkey@0"
I20260927 05:04:14.751560 9487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:14.757105 9487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:14.760871 9487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:14.761153 9487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:14.779613 9487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46103
I20260927 05:04:14.779654 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46103 every 8 connection(s)
I20260927 05:04:14.781471 9487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:14.785208 9554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:14.789402 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9487
I20260927 05:04:14.789569 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:14.790149 21157 external_mini_cluster.cc:1428] Setting key 71df68b6f03a974183b30cd8cb2517eb
I20260927 05:04:14.797746 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:14.791980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36185 (local address 127.0.0.1:46103)
0927 05:04:14.792743 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:14.792760 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:14.792801 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:04:14.794433 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:14.794454 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:14.794796 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:14.795114 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:14.796177 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:14.796182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:14.796202 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:14.796224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:14.796278 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:14.796300 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:14.796465 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:14.796617 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 05:04:14.796863 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":367,"threads_started":1}
I20260927 05:04:14.798506 9554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5: Bootstrap starting.
I20260927 05:04:14.801587 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:14.801709 9554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:14.802906 9554 log.cc:824] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:14.806246 9554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5: No bootstrap required, opened a new log
I20260927 05:04:14.809170 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:14.806744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38047 (local address 127.0.0.1:46103)
0927 05:04:14.806943 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:14.806951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:14.807235 (+ 284us) server_negotiation.cc:407] Connection header received
0927 05:04:14.807407 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:14.807411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:14.807453 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:14.807535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:14.808469 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:14.808473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:14.808476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:14.808490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:14.808526 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:14.808540 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:14.808636 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:14.808775 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 05:04:14.808877 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:04:14.814342 9554 raft_consensus.cc:348] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46103 } }
I20260927 05:04:14.814704 9554 raft_consensus.cc:374] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:14.814821 9554 raft_consensus.cc:729] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16eb8e55eb24cfbadc3e3857c226bf5, State: Initialized, Role: FOLLOWER
I20260927 05:04:14.815367 9554 consensus_queue.cc:260] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46103 } }
I20260927 05:04:14.815618 9554 raft_consensus.cc:388] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:14.815778 9554 raft_consensus.cc:482] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:14.815943 9554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:14.818040 9554 raft_consensus.cc:504] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46103 } }
I20260927 05:04:14.818496 9554 leader_election.cc:302] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d16eb8e55eb24cfbadc3e3857c226bf5; no voters:
I20260927 05:04:14.819041 9554 leader_election.cc:288] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:14.819171 9564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:14.819831 9564 raft_consensus.cc:686] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [term 1 LEADER]: Becoming Leader. State: Replica: d16eb8e55eb24cfbadc3e3857c226bf5, State: Running, Role: LEADER
I20260927 05:04:14.820405 9564 consensus_queue.cc:237] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46103 } }
I20260927 05:04:14.821247 9554 sys_catalog.cc:564] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:14.823566 9565 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46103 } } }
I20260927 05:04:14.823592 9566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d16eb8e55eb24cfbadc3e3857c226bf5. Latest consensus state: current_term: 1 leader_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16eb8e55eb24cfbadc3e3857c226bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46103 } } }
I20260927 05:04:14.824060 9565 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:14.824054 9566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:14.825611 9570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:14.832053 9570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:14.845639 9570 catalog_manager.cc:1347] Generated new cluster ID: a8615a9f5a2c4edba56161b82202dd27
I20260927 05:04:14.845772 9570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:14.861335 9570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:14.861732 9570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:14.869567 9570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d16eb8e55eb24cfbadc3e3857c226bf5: Generated new TSK 0
I20260927 05:04:14.870396 9570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:14.881356 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9487
W20260927 05:04:14.882357 9549 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36185
W20260927 05:04:14.883471 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9487: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:14.883633 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9487
2026-09-27T05:04:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:14 dist-test-slave-39f2 krb5kdc[9585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:14 dist-test-slave-39f2 krb5kdc[9585](info): set up 2 sockets
Sep 27 05:04:14 dist-test-slave-39f2 krb5kdc[9585](info): commencing operation
krb5kdc: starting...
W20260927 05:04:16.001053 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:16 dist-test-slave-39f2 krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:16.264109 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:16.596163 9598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:16.596441 9598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:16.596619 9598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:16.603995 9598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:16.604110 9598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:16.604214 9598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:16.604300 9598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:16.604360 9598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:16.614360 9598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:16.615656 9598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:16.617030 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:16.628700 9603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:16.628754 9604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:16.628775 9606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:16.629452 9598 server_base.cc:1034] running on GCE node
I20260927 05:04:16.630297 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:16.631631 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:16.632885 9598 hybrid_clock.cc:648] HybridClock initialized: now 1790485456632778 us; error 51 us; skew 500 ppm
I20260927 05:04:16.633234 9598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:16 dist-test-slave-39f2 krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:16.636317 9598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:16.637699 9598 webserver.cc:460] Webserver started at http://127.0.0.1:41629/ using document root <none> and password file <none>
I20260927 05:04:16.638334 9598 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:16.638432 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:16.638700 9598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:16.641624 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "3ead9d6acc4740a1a637f822bafc68da"
format_stamp: "Formatted at 2026-09-27 05:04:16 on dist-test-slave-39f2"
server_key: "fb135d038fbaedbcd61ca2ee83361705"
server_key_iv: "c40dfb2851ef39a8a69bbea43bf1dd9e"
server_key_version: "encryptionkey@0"
I20260927 05:04:16.642306 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "3ead9d6acc4740a1a637f822bafc68da"
format_stamp: "Formatted at 2026-09-27 05:04:16 on dist-test-slave-39f2"
server_key: "fb135d038fbaedbcd61ca2ee83361705"
server_key_iv: "c40dfb2851ef39a8a69bbea43bf1dd9e"
server_key_version: "encryptionkey@0"
I20260927 05:04:16.647102 9598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:04:16.650852 9613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:16.652153 9598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:04:16.652393 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "3ead9d6acc4740a1a637f822bafc68da"
format_stamp: "Formatted at 2026-09-27 05:04:16 on dist-test-slave-39f2"
server_key: "fb135d038fbaedbcd61ca2ee83361705"
server_key_iv: "c40dfb2851ef39a8a69bbea43bf1dd9e"
server_key_version: "encryptionkey@0"
I20260927 05:04:16.652609 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:16.678960 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:16.683398 9598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:16.683688 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:16.702840 9598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34801
I20260927 05:04:16.702885 9666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34801 every 8 connection(s)
I20260927 05:04:16.704273 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:16.708133 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:16.709478 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9598
I20260927 05:04:16.709664 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:16.710160 21157 external_mini_cluster.cc:1428] Setting key d1397729a590c796fc3688c4a91c3d2f
I20260927 05:04:16.720954 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da: Bootstrap starting.
Sep 27 05:04:16 dist-test-slave-39f2 krb5kdc[9585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:04:16.723883 9667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:16.725005 9667 log.cc:824] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:16.728106 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da: No bootstrap required, opened a new log
I20260927 05:04:16.732362 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:16.712066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47233 (local address 127.0.0.1:34801)
0927 05:04:16.712927 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:16.712939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:16.712978 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:04:16.714644 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:16.714669 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:16.715067 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:16.715476 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:16.717012 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:16.718314 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:16.719627 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:16.720289 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:16.720488 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:16.724601 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:16.724609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:16.724623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:16.724658 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:16.728135 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:16.729076 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:16.729083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:16.729089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:16.729259 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:16.729687 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:16.729690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:16.729692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:16.730252 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:16.730493 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:16.730984 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 05:04:16.731371 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":364,"threads_started":1}
I20260927 05:04:16.736019 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:16.736992 9667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ead9d6acc4740a1a637f822bafc68da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260927 05:04:16.737397 9667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:16.737484 9667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ead9d6acc4740a1a637f822bafc68da, State: Initialized, Role: FOLLOWER
I20260927 05:04:16.738089 9667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ead9d6acc4740a1a637f822bafc68da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260927 05:04:16.738358 9667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:16.738500 9667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:16.738667 9667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:16.742144 9667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ead9d6acc4740a1a637f822bafc68da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260927 05:04:16.742738 9667 leader_election.cc:302] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ead9d6acc4740a1a637f822bafc68da; no voters:
I20260927 05:04:16.743450 9667 leader_election.cc:288] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:16.743559 9677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:16.744835 9677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [term 1 LEADER]: Becoming Leader. State: Replica: 3ead9d6acc4740a1a637f822bafc68da, State: Running, Role: LEADER
I20260927 05:04:16.745395 9677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ead9d6acc4740a1a637f822bafc68da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260927 05:04:16.745734 9667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:16.756907 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ead9d6acc4740a1a637f822bafc68da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ead9d6acc4740a1a637f822bafc68da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } } }
I20260927 05:04:16.757364 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:16.760224 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ead9d6acc4740a1a637f822bafc68da. Latest consensus state: current_term: 1 leader_uuid: "3ead9d6acc4740a1a637f822bafc68da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ead9d6acc4740a1a637f822bafc68da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } } }
I20260927 05:04:16.760581 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:16.761825 9688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:16.762279 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:16.741083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58349 (local address 127.0.0.1:34801)
0927 05:04:16.741284 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:16.741293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:16.741773 (+ 480us) server_negotiation.cc:407] Connection header received
0927 05:04:16.741949 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:16.741953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:16.742010 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:16.742080 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:16.744140 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:16.745303 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:16.747057 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:16.748028 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:16.748183 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:16.752060 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:16.752070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:16.752078 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:16.752154 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:16.757042 (+ 4888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:16.758493 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:16.758501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:16.758507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:16.758704 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:16.759296 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:16.759303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:16.759309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:16.759869 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:16.761219 (+ 1350us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:16.761393 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 05:04:16.761683 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:04:16.767359 9688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:16.777164 9688 catalog_manager.cc:1347] Generated new cluster ID: 8cbefaf7ee2c421ebbca11af97f536c9
I20260927 05:04:16.777334 9688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:16.791669 9688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:16.793092 9688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:16.801285 9688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ead9d6acc4740a1a637f822bafc68da: Generated new TSK 0
I20260927 05:04:16.802146 9688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:16.807679 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9598
I20260927 05:04:16.808859 9662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:16.899104 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9598
2026-09-27T05:04:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:17 dist-test-slave-39f2 krb5kdc[9700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:17 dist-test-slave-39f2 krb5kdc[9700](info): set up 2 sockets
Sep 27 05:04:17 dist-test-slave-39f2 krb5kdc[9700](info): commencing operation
krb5kdc: starting...
W20260927 05:04:18.017120 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:18 dist-test-slave-39f2 krb5kdc[9700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:18.280934 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:18.626004 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:18.626506 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:18.626705 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:18.638311 9713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:18.638464 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:18.638558 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:18.638646 9713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:18.638710 9713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:18.658126 9713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:18.659453 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:18.660990 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:18.673583 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:18.673553 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:18.673688 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:18.674420 9713 server_base.cc:1034] running on GCE node
I20260927 05:04:18.675196 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:18.676594 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:18.677836 9713 hybrid_clock.cc:648] HybridClock initialized: now 1790485458677800 us; error 45 us; skew 500 ppm
I20260927 05:04:18.678196 9713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:18 dist-test-slave-39f2 krb5kdc[9700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:18.681352 9713 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:18.682809 9713 webserver.cc:460] Webserver started at http://127.0.0.1:41305/ using document root <none> and password file <none>
I20260927 05:04:18.683477 9713 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:18.683619 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:18.683975 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:18.686955 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "6f7f044d3189489ab23082fc2a8c886d"
format_stamp: "Formatted at 2026-09-27 05:04:18 on dist-test-slave-39f2"
server_key: "2ab39f64750eec46e829a5f3528b72c0"
server_key_iv: "4ff9b59167c2df1f9e66d39eb4bcb47d"
server_key_version: "encryptionkey@0"
I20260927 05:04:18.687722 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "6f7f044d3189489ab23082fc2a8c886d"
format_stamp: "Formatted at 2026-09-27 05:04:18 on dist-test-slave-39f2"
server_key: "2ab39f64750eec46e829a5f3528b72c0"
server_key_iv: "4ff9b59167c2df1f9e66d39eb4bcb47d"
server_key_version: "encryptionkey@0"
I20260927 05:04:18.692800 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:04:18.696195 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:18.697674 9713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:04:18.697849 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "6f7f044d3189489ab23082fc2a8c886d"
format_stamp: "Formatted at 2026-09-27 05:04:18 on dist-test-slave-39f2"
server_key: "2ab39f64750eec46e829a5f3528b72c0"
server_key_iv: "4ff9b59167c2df1f9e66d39eb4bcb47d"
server_key_version: "encryptionkey@0"
I20260927 05:04:18.698038 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:18.726166 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:18.730463 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:18.730753 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:18.749920 9713 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33267
I20260927 05:04:18.749959 9780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33267 every 8 connection(s)
I20260927 05:04:18.751351 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:18.755398 9781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:18.756204 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9713
I20260927 05:04:18.756417 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:18.757086 21157 external_mini_cluster.cc:1428] Setting key 0099b54e5f24c66cc2038fd978a158ea
I20260927 05:04:18.770310 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d: Bootstrap starting.
I20260927 05:04:18.771543 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:18.758985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54621 (local address 127.0.0.1:33267)
0927 05:04:18.759923 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:18.759940 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:18.759998 (+ 58us) server_negotiation.cc:407] Connection header received
0927 05:04:18.762256 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:18.762278 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:18.762678 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:18.763033 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:18.763707 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:18.765060 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:18.767077 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:18.767764 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:18.767920 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:18.769238 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:18.769245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:18.769258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:18.769787 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:18.769872 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:18.769895 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:18.769982 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:18.770249 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 05:04:18.770536 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":440,"threads_started":1}
I20260927 05:04:18.774031 9781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:18.775404 9781 log.cc:824] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:18.775524 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:18.778813 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d: No bootstrap required, opened a new log
I20260927 05:04:18.787967 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:18.781004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50289 (local address 127.0.0.1:33267)
0927 05:04:18.781152 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:18.781156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:18.781685 (+ 529us) server_negotiation.cc:407] Connection header received
0927 05:04:18.781888 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:18.781892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:18.781956 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:18.782036 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:18.782689 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:18.783415 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:18.785433 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:18.785933 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:18.786033 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:18.787229 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:18.787234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:18.787237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:18.787254 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:18.787289 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:18.787305 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:18.787371 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:18.787545 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 05:04:18.787614 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 05:04:18.790194 9781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7f044d3189489ab23082fc2a8c886d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33267 } }
I20260927 05:04:18.790612 9781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:18.790756 9781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7f044d3189489ab23082fc2a8c886d, State: Initialized, Role: FOLLOWER
I20260927 05:04:18.791366 9781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7f044d3189489ab23082fc2a8c886d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33267 } }
I20260927 05:04:18.791625 9781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:18.791795 9781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:18.791985 9781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:18.794425 9781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7f044d3189489ab23082fc2a8c886d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33267 } }
I20260927 05:04:18.795001 9781 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7f044d3189489ab23082fc2a8c886d; no voters:
I20260927 05:04:18.795537 9781 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:18.795670 9791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:18.796527 9791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [term 1 LEADER]: Becoming Leader. State: Replica: 6f7f044d3189489ab23082fc2a8c886d, State: Running, Role: LEADER
I20260927 05:04:18.797132 9791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7f044d3189489ab23082fc2a8c886d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33267 } }
I20260927 05:04:18.797899 9781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:18.799775 9793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7f044d3189489ab23082fc2a8c886d. Latest consensus state: current_term: 1 leader_uuid: "6f7f044d3189489ab23082fc2a8c886d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7f044d3189489ab23082fc2a8c886d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33267 } } }
I20260927 05:04:18.800218 9793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:18.800398 9792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7f044d3189489ab23082fc2a8c886d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7f044d3189489ab23082fc2a8c886d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33267 } } }
I20260927 05:04:18.800745 9792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:18.803449 9797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:18.809103 9797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:18.818020 9797 catalog_manager.cc:1347] Generated new cluster ID: 5fded6fefab948e1b09c61cd21b4ad73
I20260927 05:04:18.818128 9797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:18.840085 9797 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:18.841596 9797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:18.848493 9797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7f044d3189489ab23082fc2a8c886d: Generated new TSK 0
I20260927 05:04:18.849601 9797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:18.859874 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9713
W20260927 05:04:18.861074 9776 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54621
W20260927 05:04:18.861850 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9713: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:18.862036 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9713
2026-09-27T05:04:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:18 dist-test-slave-39f2 krb5kdc[9813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:18 dist-test-slave-39f2 krb5kdc[9813](info): set up 2 sockets
Sep 27 05:04:18 dist-test-slave-39f2 krb5kdc[9813](info): commencing operation
krb5kdc: starting...
W20260927 05:04:19.993209 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:20 dist-test-slave-39f2 krb5kdc[9813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:20.264609 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:20.602407 9826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:20.602728 9826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:20.602905 9826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:20.610339 9826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:20.610491 9826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:20.610606 9826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:20.610713 9826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:20.610795 9826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:20.621245 9826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:20.622601 9826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:20.623925 9826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:20.636580 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:20.636545 9834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:20.636605 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:20.637831 9826 server_base.cc:1034] running on GCE node
I20260927 05:04:20.638540 9826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:20.639896 9826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:20.641146 9826 hybrid_clock.cc:648] HybridClock initialized: now 1790485460641093 us; error 47 us; skew 500 ppm
I20260927 05:04:20.641512 9826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:20 dist-test-slave-39f2 krb5kdc[9813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:20.644764 9826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:20.646358 9826 webserver.cc:460] Webserver started at http://127.0.0.1:33655/ using document root <none> and password file <none>
I20260927 05:04:20.647094 9826 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:20.647250 9826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:20.647570 9826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:20.650724 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "1ef17a0ea4894a3fbf56576195feabfe"
format_stamp: "Formatted at 2026-09-27 05:04:20 on dist-test-slave-39f2"
server_key: "6cb27903346e1a2e47ff57f3b4c175e1"
server_key_iv: "854068de7a0fccd3dd2fb48c902a9419"
server_key_version: "encryptionkey@0"
I20260927 05:04:20.651556 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "1ef17a0ea4894a3fbf56576195feabfe"
format_stamp: "Formatted at 2026-09-27 05:04:20 on dist-test-slave-39f2"
server_key: "6cb27903346e1a2e47ff57f3b4c175e1"
server_key_iv: "854068de7a0fccd3dd2fb48c902a9419"
server_key_version: "encryptionkey@0"
I20260927 05:04:20.657123 9826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:04:20.660699 9841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:20.662108 9826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:04:20.662283 9826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "1ef17a0ea4894a3fbf56576195feabfe"
format_stamp: "Formatted at 2026-09-27 05:04:20 on dist-test-slave-39f2"
server_key: "6cb27903346e1a2e47ff57f3b4c175e1"
server_key_iv: "854068de7a0fccd3dd2fb48c902a9419"
server_key_version: "encryptionkey@0"
I20260927 05:04:20.662477 9826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:20.668388 9826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:20.673447 9826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:20.673738 9826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:20.691444 9826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38795
I20260927 05:04:20.691526 9892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38795 every 8 connection(s)
I20260927 05:04:20.692852 9826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:20.697187 9893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:20.700079 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9826
I20260927 05:04:20.700258 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:20.700680 21157 external_mini_cluster.cc:1428] Setting key 469853291e4430046dd57dd99eeb5fcb
I20260927 05:04:20.708534 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:20.702628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43531 (local address 127.0.0.1:38795)
0927 05:04:20.703480 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:20.703491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:20.703530 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:04:20.705204 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:20.705243 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:20.705600 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:20.705932 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:20.706999 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:20.707005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:20.707018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:20.707038 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:20.707101 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:20.707121 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:20.707331 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:20.707442 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 05:04:20.707657 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":414,"threads_started":1}
I20260927 05:04:20.709355 9893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe: Bootstrap starting.
I20260927 05:04:20.712150 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:20.712697 9893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:20.713685 9893 log.cc:824] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:20.716419 9893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe: No bootstrap required, opened a new log
I20260927 05:04:20.720731 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:20.718250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59613 (local address 127.0.0.1:38795)
0927 05:04:20.718474 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:20.718481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:20.718836 (+ 355us) server_negotiation.cc:407] Connection header received
0927 05:04:20.719012 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:20.719015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:20.719068 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:20.719164 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:20.720045 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:20.720049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:20.720052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:20.720067 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:20.720106 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:20.720121 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:20.720218 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:20.720403 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 05:04:20.720451 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:04:20.725015 9893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef17a0ea4894a3fbf56576195feabfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260927 05:04:20.725375 9893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:20.725481 9893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef17a0ea4894a3fbf56576195feabfe, State: Initialized, Role: FOLLOWER
I20260927 05:04:20.726042 9893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef17a0ea4894a3fbf56576195feabfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260927 05:04:20.726267 9893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:20.726421 9893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:20.726598 9893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:20.728945 9893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef17a0ea4894a3fbf56576195feabfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260927 05:04:20.729362 9893 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ef17a0ea4894a3fbf56576195feabfe; no voters:
I20260927 05:04:20.729843 9893 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:20.730003 9903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:20.730645 9903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [term 1 LEADER]: Becoming Leader. State: Replica: 1ef17a0ea4894a3fbf56576195feabfe, State: Running, Role: LEADER
I20260927 05:04:20.731178 9903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef17a0ea4894a3fbf56576195feabfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } }
I20260927 05:04:20.731905 9893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:20.734197 9905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef17a0ea4894a3fbf56576195feabfe. Latest consensus state: current_term: 1 leader_uuid: "1ef17a0ea4894a3fbf56576195feabfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef17a0ea4894a3fbf56576195feabfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } } }
I20260927 05:04:20.734623 9905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:20.736107 9910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:20.736006 9904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ef17a0ea4894a3fbf56576195feabfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef17a0ea4894a3fbf56576195feabfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38795 } } }
I20260927 05:04:20.736346 9904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:20.741475 9910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:20.750509 9910 catalog_manager.cc:1347] Generated new cluster ID: 1333734707dd4a64929387b7a28aeade
I20260927 05:04:20.750620 9910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:20.797358 9910 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:20.797796 9910 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:20.805878 9910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef17a0ea4894a3fbf56576195feabfe: Generated new TSK 0
I20260927 05:04:20.806787 9910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:20.825519 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9826
W20260927 05:04:20.826555 9888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43531
W20260927 05:04:20.827292 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:20.827427 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9826
2026-09-27T05:04:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:20 dist-test-slave-39f2 krb5kdc[9924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:20 dist-test-slave-39f2 krb5kdc[9924](info): set up 2 sockets
Sep 27 05:04:20 dist-test-slave-39f2 krb5kdc[9924](info): commencing operation
krb5kdc: starting...
W20260927 05:04:21.973059 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:22 dist-test-slave-39f2 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:22.244182 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:22.581213 9937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:22.581526 9937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:22.581734 9937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:22.589093 9937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:22.589231 9937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:22.589350 9937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:22.589440 9937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:22.589510 9937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:22.599730 9937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:22.601130 9937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:22.602509 9937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:22.614171 9942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:22.614218 9943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:22.614259 9945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:22.615010 9937 server_base.cc:1034] running on GCE node
I20260927 05:04:22.615883 9937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:22.617398 9937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:22.618657 9937 hybrid_clock.cc:648] HybridClock initialized: now 1790485462618611 us; error 59 us; skew 500 ppm
I20260927 05:04:22.619050 9937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:22 dist-test-slave-39f2 krb5kdc[9924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:22.622531 9937 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:22.624039 9937 webserver.cc:460] Webserver started at http://127.0.0.1:40573/ using document root <none> and password file <none>
I20260927 05:04:22.624742 9937 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:22.624918 9937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:22.625226 9937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:22.628340 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "8f89b738ecae4b04a337c0785664ce0c"
format_stamp: "Formatted at 2026-09-27 05:04:22 on dist-test-slave-39f2"
server_key: "1e81ee8a6d258383776d5999004c9e1a"
server_key_iv: "db1a8bd40a38242f41aa1e18eaee8009"
server_key_version: "encryptionkey@0"
I20260927 05:04:22.629187 9937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "8f89b738ecae4b04a337c0785664ce0c"
format_stamp: "Formatted at 2026-09-27 05:04:22 on dist-test-slave-39f2"
server_key: "1e81ee8a6d258383776d5999004c9e1a"
server_key_iv: "db1a8bd40a38242f41aa1e18eaee8009"
server_key_version: "encryptionkey@0"
I20260927 05:04:22.634368 9937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:04:22.637997 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:22.639385 9937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:04:22.639568 9937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "8f89b738ecae4b04a337c0785664ce0c"
format_stamp: "Formatted at 2026-09-27 05:04:22 on dist-test-slave-39f2"
server_key: "1e81ee8a6d258383776d5999004c9e1a"
server_key_iv: "db1a8bd40a38242f41aa1e18eaee8009"
server_key_version: "encryptionkey@0"
I20260927 05:04:22.639761 9937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:22.664342 9937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:22.668942 9937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:22.669261 9937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:22.690114 9937 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37849
I20260927 05:04:22.690124 10004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37849 every 8 connection(s)
I20260927 05:04:22.691570 9937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:22.695254 10005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:22.698949 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 9937
I20260927 05:04:22.699133 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:22.699725 21157 external_mini_cluster.cc:1428] Setting key 34abc4a0470fa9a95d4773b32a66b430
I20260927 05:04:22.708070 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c: Bootstrap starting.
I20260927 05:04:22.711838 10005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:22.713183 10005 log.cc:824] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:04:22 dist-test-slave-39f2 krb5kdc[9924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:04:22.716228 10005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c: No bootstrap required, opened a new log
I20260927 05:04:22.722801 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:22.701583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:46579 (local address 127.0.0.1:37849)
0927 05:04:22.702318 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:22.702329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:22.702369 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:04:22.703973 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:22.703995 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:22.704424 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:22.704758 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:22.706587 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:22.707923 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:22.709646 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:22.710405 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:22.710534 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:22.715443 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:22.715453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:22.715470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:22.715516 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:22.718844 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:22.719601 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:22.719607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:22.719613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:22.719759 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:22.720208 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:22.720212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:22.720214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:22.720728 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:22.721076 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:22.721517 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 05:04:22.721831 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":349,"threads_started":1}
I20260927 05:04:22.724370 10005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f89b738ecae4b04a337c0785664ce0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37849 } }
I20260927 05:04:22.724745 10005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:22.724893 10005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f89b738ecae4b04a337c0785664ce0c, State: Initialized, Role: FOLLOWER
I20260927 05:04:22.725523 10005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f89b738ecae4b04a337c0785664ce0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37849 } }
I20260927 05:04:22.725816 10005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:22.725947 10005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:22.726130 10005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:22.727056 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:22.729344 10005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f89b738ecae4b04a337c0785664ce0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37849 } }
I20260927 05:04:22.729926 10005 leader_election.cc:302] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f89b738ecae4b04a337c0785664ce0c; no voters:
I20260927 05:04:22.730680 10005 leader_election.cc:288] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:22.730863 10010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:22.732370 10010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [term 1 LEADER]: Becoming Leader. State: Replica: 8f89b738ecae4b04a337c0785664ce0c, State: Running, Role: LEADER
I20260927 05:04:22.733062 10010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f89b738ecae4b04a337c0785664ce0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37849 } }
I20260927 05:04:22.733403 10005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:22.735912 10013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f89b738ecae4b04a337c0785664ce0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f89b738ecae4b04a337c0785664ce0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37849 } } }
I20260927 05:04:22.736694 10013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:22.736251 10014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f89b738ecae4b04a337c0785664ce0c. Latest consensus state: current_term: 1 leader_uuid: "8f89b738ecae4b04a337c0785664ce0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f89b738ecae4b04a337c0785664ce0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37849 } } }
I20260927 05:04:22.737390 10014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:22.737875 10020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:22.744390 10020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:22.761199 10020 catalog_manager.cc:1347] Generated new cluster ID: acff6692eae64f75898db8c4c0472d98
I20260927 05:04:22.761317 10020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:22.767133 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:22.738029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:36359 (local address 127.0.0.1:37849)
0927 05:04:22.738194 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:22.738200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:22.749435 (+ 11235us) server_negotiation.cc:407] Connection header received
0927 05:04:22.749783 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:22.749794 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:22.749902 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:22.749997 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:22.752055 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:22.753188 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:22.755379 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:22.756322 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:22.756436 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:22.760141 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:22.760150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:22.760154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:22.760190 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:22.763859 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:22.764699 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:22.764705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:22.764708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:22.764866 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:22.765359 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:22.765364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:22.765368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:22.765761 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:22.765998 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:22.766597 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 05:04:22.766763 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:04:22.774061 10020 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:22.776373 10020 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:22.789625 10020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f89b738ecae4b04a337c0785664ce0c: Generated new TSK 0
I20260927 05:04:22.790665 10020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:22.803279 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 9937
I20260927 05:04:22.804673 10000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:22.895854 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 9937
W20260927 05:04:22.914410 9922 connection.cc:535] client connection to 127.0.0.1:37849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37849): Connection reset by peer (error 104)
2026-09-27T05:04:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:22 dist-test-slave-39f2 krb5kdc[10037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:22 dist-test-slave-39f2 krb5kdc[10037](info): set up 2 sockets
Sep 27 05:04:22 dist-test-slave-39f2 krb5kdc[10037](info): commencing operation
krb5kdc: starting...
W20260927 05:04:24.013064 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:24 dist-test-slave-39f2 krb5kdc[10037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:24.284354 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:24.624626 10050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:24.624974 10050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:24.625129 10050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:24.632611 10050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:24.632752 10050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:24.632880 10050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:24.632983 10050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:24.633054 10050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:24.643123 10050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:24.644556 10050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:24.645906 10050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:24.657519 10055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:24.657543 10058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:24.657584 10056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:24.658495 10050 server_base.cc:1034] running on GCE node
I20260927 05:04:24.659250 10050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:24.660625 10050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:24.661846 10050 hybrid_clock.cc:648] HybridClock initialized: now 1790485464661799 us; error 81 us; skew 500 ppm
I20260927 05:04:24.662190 10050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:24 dist-test-slave-39f2 krb5kdc[10037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:24.668339 10050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:24.669814 10050 webserver.cc:460] Webserver started at http://127.0.0.1:42243/ using document root <none> and password file <none>
I20260927 05:04:24.670476 10050 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:24.670614 10050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:24.670917 10050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:24.674031 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "9d41c37ba3804e44a753edc9062a1240"
format_stamp: "Formatted at 2026-09-27 05:04:24 on dist-test-slave-39f2"
server_key: "feb91adaa47dd69ec0ab239b976aea38"
server_key_iv: "fb1be3f1bbe3fdaad81abdaa0b68edcc"
server_key_version: "encryptionkey@0"
I20260927 05:04:24.674782 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "9d41c37ba3804e44a753edc9062a1240"
format_stamp: "Formatted at 2026-09-27 05:04:24 on dist-test-slave-39f2"
server_key: "feb91adaa47dd69ec0ab239b976aea38"
server_key_iv: "fb1be3f1bbe3fdaad81abdaa0b68edcc"
server_key_version: "encryptionkey@0"
I20260927 05:04:24.679801 10050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:04:24.683429 10065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:24.684762 10050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:04:24.684983 10050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "9d41c37ba3804e44a753edc9062a1240"
format_stamp: "Formatted at 2026-09-27 05:04:24 on dist-test-slave-39f2"
server_key: "feb91adaa47dd69ec0ab239b976aea38"
server_key_iv: "fb1be3f1bbe3fdaad81abdaa0b68edcc"
server_key_version: "encryptionkey@0"
I20260927 05:04:24.685181 10050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:24.712551 10050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:24.717039 10050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:24.717322 10050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:24.736136 10050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36667
I20260927 05:04:24.736163 10117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36667 every 8 connection(s)
I20260927 05:04:24.737622 10050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:24.740067 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10050
I20260927 05:04:24.740252 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:24.740859 21157 external_mini_cluster.cc:1428] Setting key d49330f08e57fcb4ea8109b1bd40c012
I20260927 05:04:24.742072 10118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:24.755204 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:24.742617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:59789 (local address 127.0.0.1:36667)
0927 05:04:24.743433 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:24.743448 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:24.743497 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:04:24.745498 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:24.745525 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:24.746050 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:24.746486 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:24.746975 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:24.749004 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:24.750619 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:24.751648 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:24.751819 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:24.753056 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:24.753063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:24.753076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:24.753611 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:24.753680 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:24.753707 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:24.753905 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:24.754172 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 05:04:24.754331 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":339,"threads_started":1}
I20260927 05:04:24.755861 10118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240: Bootstrap starting.
I20260927 05:04:24.758895 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:24.759099 10118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:24.760385 10118 log.cc:824] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:24.762919 10118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240: No bootstrap required, opened a new log
I20260927 05:04:24.770844 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:24.763844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:58581 (local address 127.0.0.1:36667)
0927 05:04:24.764070 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:24.764076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:24.764413 (+ 337us) server_negotiation.cc:407] Connection header received
0927 05:04:24.764570 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:24.764574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:24.764634 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:24.764717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:24.765368 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:24.766181 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:24.768098 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:24.768584 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:24.768685 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:24.769962 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:24.769967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:24.769970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:24.770000 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:24.770058 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:24.770074 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:24.770162 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:24.770408 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 05:04:24.770499 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:04:24.772533 10118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d41c37ba3804e44a753edc9062a1240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260927 05:04:24.772969 10118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:24.773082 10118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d41c37ba3804e44a753edc9062a1240, State: Initialized, Role: FOLLOWER
I20260927 05:04:24.773674 10118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d41c37ba3804e44a753edc9062a1240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260927 05:04:24.773902 10118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:24.774049 10118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:24.774220 10118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:24.776389 10118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d41c37ba3804e44a753edc9062a1240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260927 05:04:24.776944 10118 leader_election.cc:302] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d41c37ba3804e44a753edc9062a1240; no voters:
I20260927 05:04:24.777540 10118 leader_election.cc:288] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:24.777729 10128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:24.778443 10128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [term 1 LEADER]: Becoming Leader. State: Replica: 9d41c37ba3804e44a753edc9062a1240, State: Running, Role: LEADER
I20260927 05:04:24.779029 10128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d41c37ba3804e44a753edc9062a1240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260927 05:04:24.779747 10118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:24.781837 10129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d41c37ba3804e44a753edc9062a1240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d41c37ba3804e44a753edc9062a1240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } } }
I20260927 05:04:24.781867 10130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d41c37ba3804e44a753edc9062a1240. Latest consensus state: current_term: 1 leader_uuid: "9d41c37ba3804e44a753edc9062a1240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d41c37ba3804e44a753edc9062a1240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } } }
I20260927 05:04:24.783833 10129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:24.783917 10130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:24.785506 10134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:24.791373 10134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:24.799619 10134 catalog_manager.cc:1347] Generated new cluster ID: 3214b139a4954e928c0ab1b1623d0d14
I20260927 05:04:24.799727 10134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:24.815016 10134 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:24.816249 10134 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:24.824898 10134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d41c37ba3804e44a753edc9062a1240: Generated new TSK 0
I20260927 05:04:24.825848 10134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:24.840786 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10050
W20260927 05:04:24.842031 10113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:59789
W20260927 05:04:24.842911 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10050: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:24.843080 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10050
W20260927 05:04:24.860761 10035 connection.cc:535] client connection to 127.0.0.1:36667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36667): Connection reset by peer (error 104)
2026-09-27T05:04:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:24 dist-test-slave-39f2 krb5kdc[10149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:24 dist-test-slave-39f2 krb5kdc[10149](info): set up 2 sockets
Sep 27 05:04:24 dist-test-slave-39f2 krb5kdc[10149](info): commencing operation
krb5kdc: starting...
W20260927 05:04:25.985051 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:26 dist-test-slave-39f2 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:26.256556 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:26.599134 10162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:26.599480 10162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:26.599644 10162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:26.606719 10162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:26.606860 10162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:26.606951 10162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:26.607067 10162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:26.607141 10162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:26.617578 10162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:26.618963 10162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:26.620275 10162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:26.632457 10167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:26.632493 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:26.632956 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:26.633291 10162 server_base.cc:1034] running on GCE node
I20260927 05:04:26.634059 10162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:26.635345 10162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:26.636570 10162 hybrid_clock.cc:648] HybridClock initialized: now 1790485466636528 us; error 36 us; skew 500 ppm
I20260927 05:04:26.636996 10162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:26 dist-test-slave-39f2 krb5kdc[10149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:26.640024 10162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:26.641531 10162 webserver.cc:460] Webserver started at http://127.0.0.1:38957/ using document root <none> and password file <none>
I20260927 05:04:26.642206 10162 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:26.642351 10162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:26.642648 10162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:26.645690 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "d568120a4cd74b2ea9226b4b3c8f57cd"
format_stamp: "Formatted at 2026-09-27 05:04:26 on dist-test-slave-39f2"
server_key: "c67b67be3b4ba06b76eb941712ff9b2d"
server_key_iv: "c64c4ca2f408b2eb962408bccbe2d3dc"
server_key_version: "encryptionkey@0"
I20260927 05:04:26.646459 10162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "d568120a4cd74b2ea9226b4b3c8f57cd"
format_stamp: "Formatted at 2026-09-27 05:04:26 on dist-test-slave-39f2"
server_key: "c67b67be3b4ba06b76eb941712ff9b2d"
server_key_iv: "c64c4ca2f408b2eb962408bccbe2d3dc"
server_key_version: "encryptionkey@0"
I20260927 05:04:26.651247 10162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 05:04:26.654461 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:26.655758 10162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:04:26.655933 10162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "d568120a4cd74b2ea9226b4b3c8f57cd"
format_stamp: "Formatted at 2026-09-27 05:04:26 on dist-test-slave-39f2"
server_key: "c67b67be3b4ba06b76eb941712ff9b2d"
server_key_iv: "c64c4ca2f408b2eb962408bccbe2d3dc"
server_key_version: "encryptionkey@0"
I20260927 05:04:26.656122 10162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:26.661825 10162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:26.665719 10162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:26.666004 10162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:26.684865 10162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42423
I20260927 05:04:26.684958 10228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42423 every 8 connection(s)
I20260927 05:04:26.686270 10162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:26.690873 10229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:26.692139 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10162
I20260927 05:04:26.692350 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:26.692790 21157 external_mini_cluster.cc:1428] Setting key ec514d9411618a415cc1be3d38d5b107
I20260927 05:04:26.702353 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:26.695501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:59697 (local address 127.0.0.1:42423)
0927 05:04:26.697250 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:26.697268 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:26.697312 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:04:26.698910 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:26.698934 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:26.699299 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:26.699729 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:26.700732 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:26.700740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:26.700753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:26.700768 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:26.700869 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:26.700904 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:26.701123 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:26.701285 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 05:04:26.701469 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1424,"thread_start_us":1148,"threads_started":1}
I20260927 05:04:26.706048 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:26.706213 10229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd: Bootstrap starting.
I20260927 05:04:26.709445 10229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:26.710846 10229 log.cc:824] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:26.713671 10229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd: No bootstrap required, opened a new log
I20260927 05:04:26.714061 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:26.710384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:47825 (local address 127.0.0.1:42423)
0927 05:04:26.710532 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:26.710536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:26.712136 (+ 1600us) server_negotiation.cc:407] Connection header received
0927 05:04:26.712342 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:26.712346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:26.712392 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:26.712489 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:26.713397 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:26.713404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:26.713408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:26.713426 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:26.713468 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:26.713483 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:26.713595 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:26.713694 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 05:04:26.713762 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:04:26.720978 10229 raft_consensus.cc:348] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260927 05:04:26.721392 10229 raft_consensus.cc:374] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:26.721517 10229 raft_consensus.cc:729] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d568120a4cd74b2ea9226b4b3c8f57cd, State: Initialized, Role: FOLLOWER
I20260927 05:04:26.722034 10229 consensus_queue.cc:260] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260927 05:04:26.722261 10229 raft_consensus.cc:388] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:26.722397 10229 raft_consensus.cc:482] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:26.722564 10229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:26.724587 10229 raft_consensus.cc:504] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260927 05:04:26.725095 10229 leader_election.cc:302] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d568120a4cd74b2ea9226b4b3c8f57cd; no voters:
I20260927 05:04:26.725687 10229 leader_election.cc:288] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:26.725840 10239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:26.727308 10239 raft_consensus.cc:686] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [term 1 LEADER]: Becoming Leader. State: Replica: d568120a4cd74b2ea9226b4b3c8f57cd, State: Running, Role: LEADER
I20260927 05:04:26.727799 10229 sys_catalog.cc:564] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:26.727939 10239 consensus_queue.cc:237] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260927 05:04:26.730973 10241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d568120a4cd74b2ea9226b4b3c8f57cd. Latest consensus state: current_term: 1 leader_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260927 05:04:26.731025 10240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d568120a4cd74b2ea9226b4b3c8f57cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260927 05:04:26.731493 10240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:26.731691 10241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:26.733516 10249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:26.738346 10249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:26.747511 10249 catalog_manager.cc:1347] Generated new cluster ID: 5ed213a1638340ff967df57f90aebe68
I20260927 05:04:26.747625 10249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:26.756690 10249 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:26.757081 10249 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:26.765211 10249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d568120a4cd74b2ea9226b4b3c8f57cd: Generated new TSK 0
I20260927 05:04:26.765992 10249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:26.773622 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10162
W20260927 05:04:26.775236 10224 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:59697
W20260927 05:04:26.775926 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:26.776075 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10162
2026-09-27T05:04:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:26 dist-test-slave-39f2 krb5kdc[10261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:26 dist-test-slave-39f2 krb5kdc[10261](info): set up 2 sockets
Sep 27 05:04:26 dist-test-slave-39f2 krb5kdc[10261](info): commencing operation
krb5kdc: starting...
W20260927 05:04:27.901322 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:28 dist-test-slave-39f2 krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:28.173108 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:28.513712 10274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:28.514019 10274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:28.514249 10274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:28.521646 10274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:28.521754 10274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:28.521895 10274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:28.522023 10274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:28.522115 10274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:28.532128 10274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:28.533505 10274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:28.534926 10274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:28.547526 10280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:28.547786 10279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:28.548070 10282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:28.548344 10274 server_base.cc:1034] running on GCE node
I20260927 05:04:28.549188 10274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:28.550594 10274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:28.551805 10274 hybrid_clock.cc:648] HybridClock initialized: now 1790485468551748 us; error 39 us; skew 500 ppm
I20260927 05:04:28.552129 10274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:28 dist-test-slave-39f2 krb5kdc[10261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:28.555287 10274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:28.556690 10274 webserver.cc:460] Webserver started at http://127.0.0.1:46673/ using document root <none> and password file <none>
I20260927 05:04:28.557471 10274 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:28.557602 10274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:28.557852 10274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:28.560745 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "c990efa624834719b4492ec3386ed3ed"
format_stamp: "Formatted at 2026-09-27 05:04:28 on dist-test-slave-39f2"
server_key: "28d80012861f5f1fa9896ecc0077e377"
server_key_iv: "64f170519cd895a3bc5129adb867d084"
server_key_version: "encryptionkey@0"
I20260927 05:04:28.561508 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "c990efa624834719b4492ec3386ed3ed"
format_stamp: "Formatted at 2026-09-27 05:04:28 on dist-test-slave-39f2"
server_key: "28d80012861f5f1fa9896ecc0077e377"
server_key_iv: "64f170519cd895a3bc5129adb867d084"
server_key_version: "encryptionkey@0"
I20260927 05:04:28.566547 10274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 05:04:28.570046 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:28.571352 10274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:28.571491 10274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "c990efa624834719b4492ec3386ed3ed"
format_stamp: "Formatted at 2026-09-27 05:04:28 on dist-test-slave-39f2"
server_key: "28d80012861f5f1fa9896ecc0077e377"
server_key_iv: "64f170519cd895a3bc5129adb867d084"
server_key_version: "encryptionkey@0"
I20260927 05:04:28.571650 10274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:28.605785 10274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:28.609879 10274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:28.610121 10274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:28.628712 10274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35991
I20260927 05:04:28.628747 10340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35991 every 8 connection(s)
I20260927 05:04:28.630142 10274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:28.633891 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:28.638792 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10274
I20260927 05:04:28.638960 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:28.639394 21157 external_mini_cluster.cc:1428] Setting key 02f22a38ac35753583a344e62a5dc95d
I20260927 05:04:28.645812 10341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed: Bootstrap starting.
I20260927 05:04:28.649364 10341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:28.650435 10341 log.cc:824] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:28.653355 10341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed: No bootstrap required, opened a new log
Sep 27 05:04:28 dist-test-slave-39f2 krb5kdc[10261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:04:28.663893 10341 raft_consensus.cc:348] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260927 05:04:28.664388 10341 raft_consensus.cc:374] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:28.664535 10341 raft_consensus.cc:729] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c990efa624834719b4492ec3386ed3ed, State: Initialized, Role: FOLLOWER
I20260927 05:04:28.668385 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:28.641373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:49927 (local address 127.0.0.1:35991)
0927 05:04:28.642209 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:28.642223 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:28.642291 (+ 68us) server_negotiation.cc:407] Connection header received
0927 05:04:28.644742 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:28.644767 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:28.645300 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:28.645702 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:28.646967 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:28.649065 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:28.650378 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:28.651348 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:28.651477 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:28.656408 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:28.656418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:28.656436 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:28.656484 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:28.661201 (+ 4717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:28.662593 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:28.662634 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:28.662649 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:28.663113 (+ 464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:28.663970 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:28.663981 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:28.663988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:28.665042 (+ 1054us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:28.665445 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:28.666329 (+ 884us) server_negotiation.cc:299] Negotiation successful
0927 05:04:28.666730 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260927 05:04:28.668447 10341 consensus_queue.cc:260] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260927 05:04:28.669085 10341 raft_consensus.cc:388] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:28.669211 10341 raft_consensus.cc:482] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:28.669449 10341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:28.673429 10341 raft_consensus.cc:504] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260927 05:04:28.674077 10341 leader_election.cc:302] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c990efa624834719b4492ec3386ed3ed; no voters:
I20260927 05:04:28.674530 10346 raft_consensus.cc:482] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:04:28.674934 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:28.674974 10341 leader_election.cc:288] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:28.677047 10341 sys_catalog.cc:564] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:28.680655 10346 raft_consensus.cc:504] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260927 05:04:28.681675 10347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:28.682350 10347 raft_consensus.cc:686] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [term 1 LEADER]: Becoming Leader. State: Replica: c990efa624834719b4492ec3386ed3ed, State: Running, Role: LEADER
I20260927 05:04:28.682572 10346 leader_election.cc:302] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c990efa624834719b4492ec3386ed3ed; no voters:
I20260927 05:04:28.682993 10347 consensus_queue.cc:237] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260927 05:04:28.684855 10346 leader_election.cc:288] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260927 05:04:28.685537 10348 sys_catalog.cc:455] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c990efa624834719b4492ec3386ed3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } } }
I20260927 05:04:28.685912 10348 sys_catalog.cc:458] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:28.686465 10356 sys_catalog.cc:455] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader c990efa624834719b4492ec3386ed3ed. Latest consensus state: current_term: 1 leader_uuid: "c990efa624834719b4492ec3386ed3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990efa624834719b4492ec3386ed3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } } }
I20260927 05:04:28.686724 10356 sys_catalog.cc:458] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:28.687601 10359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:28.701172 10359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:28.716080 10359 catalog_manager.cc:1347] Generated new cluster ID: 80848e7a10934e41b5f2a6823fd42906
I20260927 05:04:28.716259 10359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:28.736907 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:28.710109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:52521 (local address 127.0.0.1:35991)
0927 05:04:28.710321 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:28.710328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:28.713042 (+ 2714us) server_negotiation.cc:407] Connection header received
0927 05:04:28.713244 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:28.713259 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:28.713348 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:28.713489 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:28.720580 (+ 7091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:28.721932 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:28.724950 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:28.725612 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:28.725712 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:28.729439 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:28.729446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:28.729451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:28.729489 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:28.733199 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:28.734153 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:28.734159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:28.734162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:28.734341 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:28.734928 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:28.734933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:28.734937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:28.735330 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:28.735606 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:28.736267 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 05:04:28.736479 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:04:28.737640 10359 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:28.739471 10359 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:28.750212 10359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c990efa624834719b4492ec3386ed3ed: Generated new TSK 0
I20260927 05:04:28.751076 10359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:28.758599 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10274
I20260927 05:04:28.759719 10336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:28.849633 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10274
W20260927 05:04:28.867992 10258 connection.cc:535] client connection to 127.0.0.1:35991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35991): Connection reset by peer (error 104)
2026-09-27T05:04:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:28 dist-test-slave-39f2 krb5kdc[10375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:28 dist-test-slave-39f2 krb5kdc[10375](info): set up 2 sockets
Sep 27 05:04:28 dist-test-slave-39f2 krb5kdc[10375](info): commencing operation
krb5kdc: starting...
W20260927 05:04:29.969029 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:30 dist-test-slave-39f2 krb5kdc[10375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:30.235917 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:30.578009 10388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:30.578348 10388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:30.578506 10388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:30.585863 10388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:30.586009 10388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:30.586109 10388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:30.586213 10388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:30.586288 10388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:30.596077 10388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:30.597450 10388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:30.598814 10388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:30.611064 10393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:30.611013 10396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:30.611025 10394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:30.611949 10388 server_base.cc:1034] running on GCE node
I20260927 05:04:30.612674 10388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:30.614084 10388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:30.615326 10388 hybrid_clock.cc:648] HybridClock initialized: now 1790485470615282 us; error 37 us; skew 500 ppm
I20260927 05:04:30.615658 10388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:30 dist-test-slave-39f2 krb5kdc[10375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:30.622001 10388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:30.623673 10388 webserver.cc:460] Webserver started at http://127.0.0.1:42515/ using document root <none> and password file <none>
I20260927 05:04:30.624413 10388 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:30.624562 10388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:30.624969 10388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:30.627995 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "cc8e8f975d7e4da38d63630b40923280"
format_stamp: "Formatted at 2026-09-27 05:04:30 on dist-test-slave-39f2"
server_key: "714724f3290bde8bd881c847cd20af24"
server_key_iv: "ac40bb5a57c44597d4d3a62e3c2e5aaf"
server_key_version: "encryptionkey@0"
I20260927 05:04:30.628750 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "cc8e8f975d7e4da38d63630b40923280"
format_stamp: "Formatted at 2026-09-27 05:04:30 on dist-test-slave-39f2"
server_key: "714724f3290bde8bd881c847cd20af24"
server_key_iv: "ac40bb5a57c44597d4d3a62e3c2e5aaf"
server_key_version: "encryptionkey@0"
I20260927 05:04:30.634063 10388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:04:30.637774 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:30.639338 10388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:30.639526 10388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "cc8e8f975d7e4da38d63630b40923280"
format_stamp: "Formatted at 2026-09-27 05:04:30 on dist-test-slave-39f2"
server_key: "714724f3290bde8bd881c847cd20af24"
server_key_iv: "ac40bb5a57c44597d4d3a62e3c2e5aaf"
server_key_version: "encryptionkey@0"
I20260927 05:04:30.639717 10388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:30.664109 10388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:30.668277 10388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:30.668579 10388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:30.688059 10388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38909
I20260927 05:04:30.688139 10455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38909 every 8 connection(s)
I20260927 05:04:30.689452 10388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:30.693961 10456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:30.696561 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10388
I20260927 05:04:30.696745 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:30.697219 21157 external_mini_cluster.cc:1428] Setting key 5b6d0ed90321f4a1f2abe26de70a850e
I20260927 05:04:30.708968 10456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280: Bootstrap starting.
I20260927 05:04:30.713734 10456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:30.714005 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:30.699092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:47589 (local address 127.0.0.1:38909)
0927 05:04:30.699949 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:30.699966 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:30.700024 (+ 58us) server_negotiation.cc:407] Connection header received
0927 05:04:30.702563 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:30.702595 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:30.703173 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:30.703646 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:30.704268 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:30.706599 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:30.707891 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:30.709080 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:30.709277 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:30.710758 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:30.710769 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:30.710801 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:30.711716 (+ 915us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:30.711851 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:30.711887 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:30.712047 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:30.712360 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 05:04:30.712557 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":322,"threads_started":1}
I20260927 05:04:30.715229 10456 log.cc:824] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:30.718982 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:30.719005 10456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280: No bootstrap required, opened a new log
I20260927 05:04:30.729157 10456 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8e8f975d7e4da38d63630b40923280" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260927 05:04:30.729744 10456 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:30.729852 10456 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8e8f975d7e4da38d63630b40923280, State: Initialized, Role: FOLLOWER
I20260927 05:04:30.730629 10456 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8e8f975d7e4da38d63630b40923280" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260927 05:04:30.730986 10456 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:30.731154 10456 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:30.731385 10456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:30.731695 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:30.724281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:42601 (local address 127.0.0.1:38909)
0927 05:04:30.724475 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:30.724481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:30.724989 (+ 508us) server_negotiation.cc:407] Connection header received
0927 05:04:30.725139 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:30.725144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:30.725236 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:30.725359 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:30.725842 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:30.727129 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:30.728598 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:30.729348 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:30.729490 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:30.730830 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:30.730835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:30.730838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:30.730860 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:30.730953 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:30.730983 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:30.731098 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:30.731341 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 05:04:30.731404 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:04:30.734501 10456 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8e8f975d7e4da38d63630b40923280" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260927 05:04:30.735044 10456 leader_election.cc:302] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8e8f975d7e4da38d63630b40923280; no voters:
I20260927 05:04:30.735715 10456 leader_election.cc:288] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:30.735880 10466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:30.736732 10466 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [term 1 LEADER]: Becoming Leader. State: Replica: cc8e8f975d7e4da38d63630b40923280, State: Running, Role: LEADER
I20260927 05:04:30.737362 10466 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8e8f975d7e4da38d63630b40923280" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } }
I20260927 05:04:30.738106 10456 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:30.740639 10468 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8e8f975d7e4da38d63630b40923280. Latest consensus state: current_term: 1 leader_uuid: "cc8e8f975d7e4da38d63630b40923280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8e8f975d7e4da38d63630b40923280" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } } }
I20260927 05:04:30.741321 10468 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:30.740603 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc8e8f975d7e4da38d63630b40923280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8e8f975d7e4da38d63630b40923280" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38909 } } }
I20260927 05:04:30.742013 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:30.744320 10473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:30.749226 10473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:30.758961 10473 catalog_manager.cc:1347] Generated new cluster ID: 0363b3b7163047399dbe96ef95061184
I20260927 05:04:30.759123 10473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:30.781581 10473 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:30.783422 10473 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:30.795922 10473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc8e8f975d7e4da38d63630b40923280: Generated new TSK 0
I20260927 05:04:30.796975 10473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:30.802489 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10388
W20260927 05:04:30.803681 10451 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:47589
W20260927 05:04:30.804553 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10388: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:30.804694 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10388
W20260927 05:04:30.823925 10373 connection.cc:535] client connection to 127.0.0.1:38909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38909): Connection reset by peer (error 104)
2026-09-27T05:04:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:30 dist-test-slave-39f2 krb5kdc[10488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:30 dist-test-slave-39f2 krb5kdc[10488](info): set up 2 sockets
Sep 27 05:04:30 dist-test-slave-39f2 krb5kdc[10488](info): commencing operation
krb5kdc: starting...
W20260927 05:04:31.925204 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:32 dist-test-slave-39f2 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:04:32.195878 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:32.527470 10501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:32.527778 10501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:32.528014 10501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:32.535384 10501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:32.535501 10501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:32.535619 10501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:32.535732 10501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:32.535794 10501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:32.545696 10501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:32.547021 10501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:32.548333 10501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:32.560211 10506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:32.560170 10507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:32.560719 10509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:32.561169 10501 server_base.cc:1034] running on GCE node
I20260927 05:04:32.561950 10501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:32.563432 10501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:32.564682 10501 hybrid_clock.cc:648] HybridClock initialized: now 1790485472564646 us; error 61 us; skew 500 ppm
I20260927 05:04:32.565093 10501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:32 dist-test-slave-39f2 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:32.568224 10501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:04:32.569871 10501 webserver.cc:460] Webserver started at http://127.0.0.1:33239/ using document root <none> and password file <none>
I20260927 05:04:32.570569 10501 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:32.570714 10501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:32.571031 10501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:32.574437 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "c662ee197d2d4f94878fd657a7701463"
format_stamp: "Formatted at 2026-09-27 05:04:32 on dist-test-slave-39f2"
server_key: "e47ca5384e764b867a9da55666a4e125"
server_key_iv: "2820bb2d42a933fd73dd4278c8f5d8a6"
server_key_version: "encryptionkey@0"
I20260927 05:04:32.575212 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "c662ee197d2d4f94878fd657a7701463"
format_stamp: "Formatted at 2026-09-27 05:04:32 on dist-test-slave-39f2"
server_key: "e47ca5384e764b867a9da55666a4e125"
server_key_iv: "2820bb2d42a933fd73dd4278c8f5d8a6"
server_key_version: "encryptionkey@0"
I20260927 05:04:32.580353 10501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:04:32.584122 10516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:32.585513 10501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:04:32.585734 10501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "c662ee197d2d4f94878fd657a7701463"
format_stamp: "Formatted at 2026-09-27 05:04:32 on dist-test-slave-39f2"
server_key: "e47ca5384e764b867a9da55666a4e125"
server_key_iv: "2820bb2d42a933fd73dd4278c8f5d8a6"
server_key_version: "encryptionkey@0"
I20260927 05:04:32.585940 10501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:32.591722 10501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:32.595817 10501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:32.596118 10501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:32.615525 10501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39057
I20260927 05:04:32.615612 10568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39057 every 8 connection(s)
I20260927 05:04:32.617046 10501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:32.620857 10569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:32.622442 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10501
I20260927 05:04:32.622622 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:32.623195 21157 external_mini_cluster.cc:1428] Setting key ce568f12645c61ac50b78f7c4c8ecb0f
I20260927 05:04:32.631249 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:32.625180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:40271 (local address 127.0.0.1:39057)
0927 05:04:32.625994 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:32.626006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:32.626049 (+ 43us) server_negotiation.cc:407] Connection header received
0927 05:04:32.627650 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:32.627672 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:32.628033 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:32.628368 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:32.629602 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:32.629609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:32.629623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:32.629637 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:32.629697 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:32.629720 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:32.629906 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:32.630083 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 05:04:32.630292 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":362,"threads_started":1}
I20260927 05:04:32.635232 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:32.635931 10569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463: Bootstrap starting.
I20260927 05:04:32.640036 10569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:32.641507 10569 log.cc:824] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:32.642700 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:32.639802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:53497 (local address 127.0.0.1:39057)
0927 05:04:32.639979 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:32.639983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:32.640571 (+ 588us) server_negotiation.cc:407] Connection header received
0927 05:04:32.640775 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:32.640780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:32.640912 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:32.641016 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:32.642014 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:32.642019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:32.642024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:32.642041 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:32.642091 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:32.642106 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:32.642221 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:32.642346 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 05:04:32.642415 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:04:32.645253 10569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463: No bootstrap required, opened a new log
I20260927 05:04:32.654114 10569 raft_consensus.cc:348] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c662ee197d2d4f94878fd657a7701463" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260927 05:04:32.654556 10569 raft_consensus.cc:374] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:32.654692 10569 raft_consensus.cc:729] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c662ee197d2d4f94878fd657a7701463, State: Initialized, Role: FOLLOWER
I20260927 05:04:32.655355 10569 consensus_queue.cc:260] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c662ee197d2d4f94878fd657a7701463" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260927 05:04:32.655637 10569 raft_consensus.cc:388] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:32.655838 10569 raft_consensus.cc:482] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:32.656036 10569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:32.658772 10569 raft_consensus.cc:504] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c662ee197d2d4f94878fd657a7701463" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260927 05:04:32.659328 10569 leader_election.cc:302] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c662ee197d2d4f94878fd657a7701463; no voters:
I20260927 05:04:32.659978 10569 leader_election.cc:288] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:32.660135 10579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:32.660873 10579 raft_consensus.cc:686] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [term 1 LEADER]: Becoming Leader. State: Replica: c662ee197d2d4f94878fd657a7701463, State: Running, Role: LEADER
I20260927 05:04:32.661480 10579 consensus_queue.cc:237] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c662ee197d2d4f94878fd657a7701463" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260927 05:04:32.662441 10569 sys_catalog.cc:564] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:32.664503 10580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c662ee197d2d4f94878fd657a7701463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c662ee197d2d4f94878fd657a7701463" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } } }
I20260927 05:04:32.665078 10580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:32.664490 10581 sys_catalog.cc:455] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c662ee197d2d4f94878fd657a7701463. Latest consensus state: current_term: 1 leader_uuid: "c662ee197d2d4f94878fd657a7701463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c662ee197d2d4f94878fd657a7701463" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } } }
I20260927 05:04:32.666316 10581 sys_catalog.cc:458] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:32.669605 10586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:32.675508 10586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:32.685305 10586 catalog_manager.cc:1347] Generated new cluster ID: 2985ded4644147c599018a17f75ebfe7
I20260927 05:04:32.685447 10586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:32.697454 10586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:32.697956 10586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:32.707500 10586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c662ee197d2d4f94878fd657a7701463: Generated new TSK 0
I20260927 05:04:32.708446 10586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:32.719954 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10501
W20260927 05:04:32.721535 10564 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.70:40271
W20260927 05:04:32.722342 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10501: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:32.722530 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10501
2026-09-27T05:04:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:32 dist-test-slave-39f2 krb5kdc[10600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:32 dist-test-slave-39f2 krb5kdc[10600](info): set up 2 sockets
Sep 27 05:04:32 dist-test-slave-39f2 krb5kdc[10600](info): commencing operation
krb5kdc: starting...
W20260927 05:04:33.912879 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:34 dist-test-slave-39f2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:34.180876 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45101
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:34.510447 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:34.510772 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:34.510943 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:34.518054 10613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:34.518218 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:34.518317 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:34.518431 10613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:34.518515 10613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:34.528270 10613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:34.529698 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:34.531028 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:34.542927 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:34.542860 10621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:34.542973 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:34.544314 10613 server_base.cc:1034] running on GCE node
I20260927 05:04:34.545153 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:34.546721 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:34.547992 10613 hybrid_clock.cc:648] HybridClock initialized: now 1790485474547943 us; error 40 us; skew 500 ppm
I20260927 05:04:34.548463 10613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:34 dist-test-slave-39f2 krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485474, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:34.551960 10613 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:34.553833 10613 webserver.cc:460] Webserver started at http://127.20.169.126:36249/ using document root <none> and password file <none>
I20260927 05:04:34.554597 10613 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:34.554759 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:34.555088 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:34.558676 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9"
format_stamp: "Formatted at 2026-09-27 05:04:34 on dist-test-slave-39f2"
server_key: "ec118a0ad771d5636e53fb15c732be7b"
server_key_iv: "bc5882f650cf8866543f450b9f5123e4"
server_key_version: "encryptionkey@0"
I20260927 05:04:34.559815 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9"
format_stamp: "Formatted at 2026-09-27 05:04:34 on dist-test-slave-39f2"
server_key: "ec118a0ad771d5636e53fb15c732be7b"
server_key_iv: "bc5882f650cf8866543f450b9f5123e4"
server_key_version: "encryptionkey@0"
I20260927 05:04:34.565514 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:04:34.569450 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:34.570904 10613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:04:34.571097 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9"
format_stamp: "Formatted at 2026-09-27 05:04:34 on dist-test-slave-39f2"
server_key: "ec118a0ad771d5636e53fb15c732be7b"
server_key_iv: "bc5882f650cf8866543f450b9f5123e4"
server_key_version: "encryptionkey@0"
I20260927 05:04:34.571300 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:34.593896 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:34.598248 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:34.598541 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:34.618148 10613 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45101
I20260927 05:04:34.618152 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45101 every 8 connection(s)
I20260927 05:04:34.619540 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:34.623163 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:34.627082 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10613
I20260927 05:04:34.627272 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:34.627821 21157 external_mini_cluster.cc:1428] Setting key c63ba020fd5bff494479d13fed189451
I20260927 05:04:34.635480 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9: Bootstrap starting.
I20260927 05:04:34.638588 10681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:34.639593 10681 log.cc:824] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:04:34 dist-test-slave-39f2 krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:34.642485 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9: No bootstrap required, opened a new log
I20260927 05:04:34.650548 10681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45101 } }
I20260927 05:04:34.650954 10681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:34.651083 10681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bffbd1c8a7d4ffe88c69be8d5bb92b9, State: Initialized, Role: FOLLOWER
I20260927 05:04:34.651507 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:34.629629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57491 (local address 127.20.169.126:45101)
0927 05:04:34.630397 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:34.630408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:34.630449 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:04:34.632059 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:34.632083 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:34.632505 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:34.632940 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:34.634485 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:34.635917 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:34.637260 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:34.637938 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:34.638148 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:34.642340 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:34.642348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:34.642362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:34.642401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:34.646122 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:34.647156 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:34.647163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:34.647169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:34.647376 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:34.648012 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:34.648021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:34.648023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:34.648677 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:34.649110 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:34.649733 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 05:04:34.650115 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":353,"threads_started":1}
I20260927 05:04:34.651685 10681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45101 } }
I20260927 05:04:34.651974 10681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:34.652135 10681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:34.652343 10681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:34.655369 10681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45101 } }
I20260927 05:04:34.655915 10681 leader_election.cc:302] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bffbd1c8a7d4ffe88c69be8d5bb92b9; no voters:
I20260927 05:04:34.656052 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:34.656602 10681 leader_election.cc:288] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:34.656769 10686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:34.657472 10686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [term 1 LEADER]: Becoming Leader. State: Replica: 6bffbd1c8a7d4ffe88c69be8d5bb92b9, State: Running, Role: LEADER
I20260927 05:04:34.658161 10686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45101 } }
I20260927 05:04:34.662951 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45101 } } }
I20260927 05:04:34.663514 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:34.667816 10686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bffbd1c8a7d4ffe88c69be8d5bb92b9. Latest consensus state: current_term: 1 leader_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bffbd1c8a7d4ffe88c69be8d5bb92b9" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45101 } } }
I20260927 05:04:34.668469 10686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:34.673205 10681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:34.673913 10693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:34.680800 10693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:34.695463 10693 catalog_manager.cc:1347] Generated new cluster ID: 5a89103a296e4934be0dc2cc38d00798
I20260927 05:04:34.695588 10693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:34.706775 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:34.673874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34951 (local address 127.20.169.126:45101)
0927 05:04:34.674105 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:34.674120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:34.684029 (+ 9909us) server_negotiation.cc:407] Connection header received
0927 05:04:34.684637 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:34.684646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:34.684763 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:34.684891 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:34.687567 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:34.689218 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:34.692065 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:34.692962 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:34.693125 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:34.697042 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:34.697052 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:34.697061 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:34.697116 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:34.701956 (+ 4840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:34.703188 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:34.703197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:34.703207 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:34.703412 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:34.704152 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:34.704160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:34.704165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:34.704737 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:34.705098 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:34.705977 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 05:04:34.706227 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:04:34.744935 10693 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:34.746330 10693 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:34.754150 10693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bffbd1c8a7d4ffe88c69be8d5bb92b9: Generated new TSK 0
I20260927 05:04:34.754989 10693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:34.769418 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10613
I20260927 05:04:34.770653 10676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:34.867506 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10613
W20260927 05:04:34.886242 10598 connection.cc:535] client connection to 127.20.169.126:45101 recv error: Network error: failed to read from TLS socket (remote: 127.20.169.126:45101): Connection reset by peer (error 104)
2026-09-27T05:04:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:34 dist-test-slave-39f2 krb5kdc[10714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:34 dist-test-slave-39f2 krb5kdc[10714](info): set up 2 sockets
Sep 27 05:04:34 dist-test-slave-39f2 krb5kdc[10714](info): commencing operation
krb5kdc: starting...
W20260927 05:04:35.969717 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:36 dist-test-slave-39f2 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:36.237146 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:40559
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:40559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:36.565026 10727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:36.565305 10727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:36.565487 10727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:36.572664 10727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:36.572777 10727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:36.572877 10727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:36.572993 10727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:36.573076 10727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:36.582818 10727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:40559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:40559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:36.584112 10727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:36.585476 10727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:36.597469 10732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:36.597433 10733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:36.597393 10735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:36.598654 10727 server_base.cc:1034] running on GCE node
I20260927 05:04:36.599507 10727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:36.601441 10727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:36.602720 10727 hybrid_clock.cc:648] HybridClock initialized: now 1790485476602674 us; error 55 us; skew 500 ppm
I20260927 05:04:36.603149 10727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:36 dist-test-slave-39f2 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485476, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:36.609922 10727 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:36.611564 10727 webserver.cc:460] Webserver started at http://127.20.169.126:37023/ using document root <none> and password file <none>
I20260927 05:04:36.612238 10727 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:36.612383 10727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:36.612703 10727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:36.615998 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "d1693206add041d6bf23975430599cd3"
format_stamp: "Formatted at 2026-09-27 05:04:36 on dist-test-slave-39f2"
server_key: "9188108926e1c4dc223976a9a081202e"
server_key_iv: "350e457e727a832044a6557308dfcae6"
server_key_version: "encryptionkey@0"
I20260927 05:04:36.616853 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "d1693206add041d6bf23975430599cd3"
format_stamp: "Formatted at 2026-09-27 05:04:36 on dist-test-slave-39f2"
server_key: "9188108926e1c4dc223976a9a081202e"
server_key_iv: "350e457e727a832044a6557308dfcae6"
server_key_version: "encryptionkey@0"
I20260927 05:04:36.621997 10727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:04:36.625447 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:36.626863 10727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:36.627045 10727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "d1693206add041d6bf23975430599cd3"
format_stamp: "Formatted at 2026-09-27 05:04:36 on dist-test-slave-39f2"
server_key: "9188108926e1c4dc223976a9a081202e"
server_key_iv: "350e457e727a832044a6557308dfcae6"
server_key_version: "encryptionkey@0"
I20260927 05:04:36.627240 10727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:36.650559 10727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:36.654830 10727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:36.655109 10727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:36.673436 10727 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:40559
I20260927 05:04:36.673470 10794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:40559 every 8 connection(s)
I20260927 05:04:36.674775 10727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:36.678700 10795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:36.682401 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10727
I20260927 05:04:36.682598 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:36.683022 21157 external_mini_cluster.cc:1428] Setting key bba23aa30ccbeef608135c838aab0a04
I20260927 05:04:36.692072 10795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3: Bootstrap starting.
I20260927 05:04:36.695971 10795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:36.697405 10795 log.cc:824] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:36.698472 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:36.684928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36333 (local address 127.20.169.126:40559)
0927 05:04:36.685675 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:36.685690 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:36.685738 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:04:36.687634 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:36.687664 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:36.688222 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:36.688667 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:36.689228 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:36.691217 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:36.692862 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:36.693918 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:36.694109 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:36.695504 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:36.695515 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:36.695534 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:36.696337 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:36.696459 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:36.696498 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:36.696658 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:36.696990 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 05:04:36.697187 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":310,"threads_started":1}
I20260927 05:04:36.700395 10795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3: No bootstrap required, opened a new log
I20260927 05:04:36.702699 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:36.709134 10795 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1693206add041d6bf23975430599cd3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40559 } }
I20260927 05:04:36.709561 10795 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:36.709666 10795 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1693206add041d6bf23975430599cd3, State: Initialized, Role: FOLLOWER
I20260927 05:04:36.710268 10795 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1693206add041d6bf23975430599cd3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40559 } }
I20260927 05:04:36.710539 10795 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:36.710700 10795 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:36.710863 10795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:36.713982 10795 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1693206add041d6bf23975430599cd3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40559 } }
I20260927 05:04:36.714591 10795 leader_election.cc:302] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1693206add041d6bf23975430599cd3; no voters:
I20260927 05:04:36.715235 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:36.707492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39073 (local address 127.20.169.126:40559)
0927 05:04:36.707742 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:36.707749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:36.708099 (+ 350us) server_negotiation.cc:407] Connection header received
0927 05:04:36.708343 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:36.708347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:36.708409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:36.708493 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:36.709153 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:36.710328 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:36.712285 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:36.712798 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:36.712979 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:36.714168 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:36.714177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:36.714183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:36.714210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:36.714330 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:36.714351 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:36.714446 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:36.714712 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 05:04:36.714784 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:04:36.715372 10795 leader_election.cc:288] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:36.715538 10805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:36.716693 10805 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [term 1 LEADER]: Becoming Leader. State: Replica: d1693206add041d6bf23975430599cd3, State: Running, Role: LEADER
I20260927 05:04:36.717289 10805 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1693206add041d6bf23975430599cd3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40559 } }
I20260927 05:04:36.717895 10795 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:36.720538 10806 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1693206add041d6bf23975430599cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1693206add041d6bf23975430599cd3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40559 } } }
I20260927 05:04:36.721027 10806 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:36.720979 10807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1693206add041d6bf23975430599cd3. Latest consensus state: current_term: 1 leader_uuid: "d1693206add041d6bf23975430599cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1693206add041d6bf23975430599cd3" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40559 } } }
I20260927 05:04:36.721369 10807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:36.725744 10811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:36.731974 10811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:36.742304 10811 catalog_manager.cc:1347] Generated new cluster ID: d7d0d439764b487f9d48c487530e1010
I20260927 05:04:36.742420 10811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:36.757000 10811 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:36.758975 10811 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:36.770123 10811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1693206add041d6bf23975430599cd3: Generated new TSK 0
I20260927 05:04:36.771212 10811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:36.777202 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10727
W20260927 05:04:36.778349 10790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36333
W20260927 05:04:36.779043 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:36.779203 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10727
W20260927 05:04:36.797259 10712 connection.cc:535] client connection to 127.20.169.126:40559 recv error: Network error: failed to read from TLS socket (remote: 127.20.169.126:40559): Connection reset by peer (error 104)
2026-09-27T05:04:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:36 dist-test-slave-39f2 krb5kdc[10826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:36 dist-test-slave-39f2 krb5kdc[10826](info): set up 2 sockets
Sep 27 05:04:36 dist-test-slave-39f2 krb5kdc[10826](info): commencing operation
krb5kdc: starting...
W20260927 05:04:37.889024 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:38 dist-test-slave-39f2 krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:38.164304 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35437
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:35437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:38.496043 10839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:38.496340 10839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:38.496438 10839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:38.503671 10839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:38.503772 10839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:38.503818 10839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:38.503882 10839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:38.503927 10839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:38.513592 10839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:35437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:38.514827 10839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:38.516134 10839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:38.527733 10844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:38.527774 10847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:38.528249 10845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:38.528658 10839 server_base.cc:1034] running on GCE node
I20260927 05:04:38.529486 10839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:38.530860 10839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:38.532066 10839 hybrid_clock.cc:648] HybridClock initialized: now 1790485478532032 us; error 26 us; skew 500 ppm
I20260927 05:04:38.532421 10839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:38 dist-test-slave-39f2 krb5kdc[10826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485478, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:38.538913 10839 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:38.540405 10839 webserver.cc:460] Webserver started at http://127.20.169.126:33871/ using document root <none> and password file <none>
I20260927 05:04:38.541152 10839 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:38.541285 10839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:38.541607 10839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:38.544589 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "1dd2962f1c804fdb80a65e4e7a3ab250"
format_stamp: "Formatted at 2026-09-27 05:04:38 on dist-test-slave-39f2"
server_key: "70e0d7ddca082c86053ffa19f3dcc76e"
server_key_iv: "c46460247e68a3e1911fc791364f7dd8"
server_key_version: "encryptionkey@0"
I20260927 05:04:38.545454 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "1dd2962f1c804fdb80a65e4e7a3ab250"
format_stamp: "Formatted at 2026-09-27 05:04:38 on dist-test-slave-39f2"
server_key: "70e0d7ddca082c86053ffa19f3dcc76e"
server_key_iv: "c46460247e68a3e1911fc791364f7dd8"
server_key_version: "encryptionkey@0"
I20260927 05:04:38.550647 10839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:04:38.554199 10854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:38.555688 10839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:04:38.555899 10839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "1dd2962f1c804fdb80a65e4e7a3ab250"
format_stamp: "Formatted at 2026-09-27 05:04:38 on dist-test-slave-39f2"
server_key: "70e0d7ddca082c86053ffa19f3dcc76e"
server_key_iv: "c46460247e68a3e1911fc791364f7dd8"
server_key_version: "encryptionkey@0"
I20260927 05:04:38.556104 10839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:38.562026 10839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:38.566138 10839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:38.566434 10839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:38.585912 10839 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:35437
I20260927 05:04:38.585942 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:35437 every 8 connection(s)
I20260927 05:04:38.587252 10839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:38.590914 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:38.595498 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10839
I20260927 05:04:38.595678 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:38.596202 21157 external_mini_cluster.cc:1428] Setting key 5acafdf7e02206ac2f15d033d9f6ed44
I20260927 05:04:38.602210 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250: Bootstrap starting.
I20260927 05:04:38.604395 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:38.597959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45853 (local address 127.20.169.126:35437)
0927 05:04:38.598908 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:38.598920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:38.598968 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:04:38.600960 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:38.600983 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:38.601338 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:38.601658 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:38.602708 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:38.602714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:38.602727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:38.602743 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:38.602809 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:38.602830 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:38.603007 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:38.603141 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 05:04:38.603368 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":447,"threads_started":1}
I20260927 05:04:38.605782 10906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:38.606882 10906 log.cc:824] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:38.608500 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:38.609724 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250: No bootstrap required, opened a new log
I20260927 05:04:38.615790 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:38.613071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53053 (local address 127.20.169.126:35437)
0927 05:04:38.613370 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:38.613376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:38.613606 (+ 230us) server_negotiation.cc:407] Connection header received
0927 05:04:38.613822 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:38.613829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:38.613894 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:38.614032 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:38.615026 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:38.615032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:38.615037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:38.615060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:38.615110 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:38.615126 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:38.615284 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:38.615397 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 05:04:38.615458 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:04:38.617986 10906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35437 } }
I20260927 05:04:38.618353 10906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:38.618422 10906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd2962f1c804fdb80a65e4e7a3ab250, State: Initialized, Role: FOLLOWER
I20260927 05:04:38.618932 10906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35437 } }
I20260927 05:04:38.619143 10906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:38.619236 10906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:38.619361 10906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:38.621665 10906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35437 } }
I20260927 05:04:38.622143 10906 leader_election.cc:302] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dd2962f1c804fdb80a65e4e7a3ab250; no voters:
I20260927 05:04:38.622701 10906 leader_election.cc:288] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:38.622872 10917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:38.623935 10917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [term 1 LEADER]: Becoming Leader. State: Replica: 1dd2962f1c804fdb80a65e4e7a3ab250, State: Running, Role: LEADER
I20260927 05:04:38.624614 10917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35437 } }
I20260927 05:04:38.625026 10906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:38.627846 10918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35437 } } }
I20260927 05:04:38.628293 10918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:38.629244 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dd2962f1c804fdb80a65e4e7a3ab250. Latest consensus state: current_term: 1 leader_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd2962f1c804fdb80a65e4e7a3ab250" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 35437 } } }
I20260927 05:04:38.629801 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:38.638069 10926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:38.643814 10926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:38.652741 10926 catalog_manager.cc:1347] Generated new cluster ID: 79e247a81f454351ab52f7d22ea752cc
I20260927 05:04:38.652877 10926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:38.669061 10926 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:38.669494 10926 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:38.685086 10926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dd2962f1c804fdb80a65e4e7a3ab250: Generated new TSK 0
I20260927 05:04:38.686043 10926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:38.703358 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10839
W20260927 05:04:38.704592 10901 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45853
W20260927 05:04:38.705540 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10839: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:38.705703 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10839
2026-09-27T05:04:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:38 dist-test-slave-39f2 krb5kdc[10939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:38 dist-test-slave-39f2 krb5kdc[10939](info): set up 2 sockets
Sep 27 05:04:38 dist-test-slave-39f2 krb5kdc[10939](info): commencing operation
krb5kdc: starting...
W20260927 05:04:39.829351 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:40 dist-test-slave-39f2 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:40.099468 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:46607
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:40.429761 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:40.430045 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:40.430140 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:40.437111 10952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:40.437217 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:40.437288 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:40.437352 10952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:40.437397 10952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:40.447026 10952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:40.448179 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:40.449481 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:40.461659 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:40.461565 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:40.461671 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:40.462515 10952 server_base.cc:1034] running on GCE node
I20260927 05:04:40.463400 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:40.464943 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:40.466192 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790485480466139 us; error 50 us; skew 500 ppm
I20260927 05:04:40.466639 10952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:40 dist-test-slave-39f2 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485480, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:40.470336 10952 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:40.471961 10952 webserver.cc:460] Webserver started at http://127.20.169.126:37287/ using document root <none> and password file <none>
I20260927 05:04:40.472725 10952 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:40.472934 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:40.473284 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:40.476908 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "1e8282cba133404085150cd4f6e4a8df"
format_stamp: "Formatted at 2026-09-27 05:04:40 on dist-test-slave-39f2"
server_key: "f9c372a70334db59e9612f2c6571a28c"
server_key_iv: "bdb3a959ae5d29a9a689e6ae67e69602"
server_key_version: "encryptionkey@0"
I20260927 05:04:40.477773 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "1e8282cba133404085150cd4f6e4a8df"
format_stamp: "Formatted at 2026-09-27 05:04:40 on dist-test-slave-39f2"
server_key: "f9c372a70334db59e9612f2c6571a28c"
server_key_iv: "bdb3a959ae5d29a9a689e6ae67e69602"
server_key_version: "encryptionkey@0"
I20260927 05:04:40.483539 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:04:40.487458 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:40.488924 10952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:04:40.489118 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "1e8282cba133404085150cd4f6e4a8df"
format_stamp: "Formatted at 2026-09-27 05:04:40 on dist-test-slave-39f2"
server_key: "f9c372a70334db59e9612f2c6571a28c"
server_key_iv: "bdb3a959ae5d29a9a689e6ae67e69602"
server_key_version: "encryptionkey@0"
I20260927 05:04:40.489318 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:40.502174 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:40.506016 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:40.506286 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:40.526679 10952 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:46607
I20260927 05:04:40.526712 11018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:46607 every 8 connection(s)
I20260927 05:04:40.528064 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:40.531838 11019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:40.534247 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 10952
I20260927 05:04:40.534430 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:40.534991 21157 external_mini_cluster.cc:1428] Setting key d3e9588d291ef173c34b05064f5b88a6
I20260927 05:04:40.545235 11019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df: Bootstrap starting.
I20260927 05:04:40.549175 11019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:40.550391 11019 log.cc:824] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:04:40 dist-test-slave-39f2 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:40.554244 11019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df: No bootstrap required, opened a new log
I20260927 05:04:40.560081 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:40.536907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45537 (local address 127.20.169.126:46607)
0927 05:04:40.537824 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:40.537836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:40.537879 (+ 43us) server_negotiation.cc:407] Connection header received
0927 05:04:40.539654 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:40.539677 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:40.540085 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:40.540774 (+ 689us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:40.542767 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:40.544109 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:40.546063 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:40.547199 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:40.547431 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:40.552234 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:40.552245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:40.552264 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:40.552372 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:40.555876 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:40.556625 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:40.556632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:40.556638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:40.556888 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:40.557279 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:40.557284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:40.557289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:40.557839 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:40.558207 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:40.558640 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 05:04:40.559067 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":462,"threads_started":1}
I20260927 05:04:40.562279 11019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8282cba133404085150cd4f6e4a8df" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46607 } }
I20260927 05:04:40.562718 11019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:40.562829 11019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8282cba133404085150cd4f6e4a8df, State: Initialized, Role: FOLLOWER
I20260927 05:04:40.563463 11019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8282cba133404085150cd4f6e4a8df" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46607 } }
I20260927 05:04:40.563756 11019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:40.563913 11019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:40.563954 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:40.564092 11019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:40.566380 11019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8282cba133404085150cd4f6e4a8df" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46607 } }
I20260927 05:04:40.566955 11019 leader_election.cc:302] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e8282cba133404085150cd4f6e4a8df; no voters:
I20260927 05:04:40.567656 11019 leader_election.cc:288] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:40.567756 11028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:40.569403 11028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [term 1 LEADER]: Becoming Leader. State: Replica: 1e8282cba133404085150cd4f6e4a8df, State: Running, Role: LEADER
I20260927 05:04:40.570116 11019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:40.569993 11028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8282cba133404085150cd4f6e4a8df" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46607 } }
I20260927 05:04:40.577876 11029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e8282cba133404085150cd4f6e4a8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8282cba133404085150cd4f6e4a8df" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46607 } } }
I20260927 05:04:40.578434 11029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:40.577939 11031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e8282cba133404085150cd4f6e4a8df. Latest consensus state: current_term: 1 leader_uuid: "1e8282cba133404085150cd4f6e4a8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8282cba133404085150cd4f6e4a8df" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46607 } } }
I20260927 05:04:40.579151 11031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:40.585870 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:40.568649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50763 (local address 127.20.169.126:46607)
0927 05:04:40.568816 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:40.568851 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:40.569294 (+ 443us) server_negotiation.cc:407] Connection header received
0927 05:04:40.569576 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:40.569581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:40.569657 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:40.569727 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:40.571415 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:40.572722 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:40.574178 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:40.574952 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:40.575084 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:40.578226 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:40.578232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:40.578236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:40.578287 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:40.582724 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:40.583524 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:40.583528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:40.583530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:40.583651 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:40.584124 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:40.584134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:40.584137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:40.584608 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:40.584908 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:40.585268 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 05:04:40.585487 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 05:04:40.586566 11043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:04:40.586683 11043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:04:40.586819 11045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:40.589246 11045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:40.598155 11045 catalog_manager.cc:1347] Generated new cluster ID: 3935e3d91aa346aa8e7cacdb4a9cbe00
I20260927 05:04:40.598256 11045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:40.615721 11045 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:40.617260 11045 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:40.633111 11045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e8282cba133404085150cd4f6e4a8df: Generated new TSK 0
I20260927 05:04:40.634162 11045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:40.649348 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 10952
I20260927 05:04:40.651261 11014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:40.744843 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 10952
2026-09-27T05:04:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:40 dist-test-slave-39f2 krb5kdc[11051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:40 dist-test-slave-39f2 krb5kdc[11051](info): set up 2 sockets
Sep 27 05:04:40 dist-test-slave-39f2 krb5kdc[11051](info): commencing operation
krb5kdc: starting...
W20260927 05:04:41.910045 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:42 dist-test-slave-39f2 krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:42.179955 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45891
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:45891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:42.521646 11065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:42.521997 11065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:42.522189 11065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:42.529397 11065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:42.529562 11065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:42.529685 11065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:42.529819 11065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:42.529918 11065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:42.539788 11065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:45891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:45891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:42.541194 11065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:42.542598 11065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:42.555330 11070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:42.555454 11073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:42.555496 11071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:42.556288 11065 server_base.cc:1034] running on GCE node
I20260927 05:04:42.557159 11065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:42.558570 11065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:42.559926 11065 hybrid_clock.cc:648] HybridClock initialized: now 1790485482559856 us; error 57 us; skew 500 ppm
I20260927 05:04:42.560405 11065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:42 dist-test-slave-39f2 krb5kdc[11051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485482, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:42.567169 11065 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:42.568801 11065 webserver.cc:460] Webserver started at http://127.20.169.126:35699/ using document root <none> and password file <none>
I20260927 05:04:42.569584 11065 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:42.569693 11065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:42.569967 11065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:42.573259 11065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea"
format_stamp: "Formatted at 2026-09-27 05:04:42 on dist-test-slave-39f2"
server_key: "47bd2a16c739106ef338f9aca19fe695"
server_key_iv: "fd96e48ace3221b2b916e24547634d30"
server_key_version: "encryptionkey@0"
I20260927 05:04:42.574043 11065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea"
format_stamp: "Formatted at 2026-09-27 05:04:42 on dist-test-slave-39f2"
server_key: "47bd2a16c739106ef338f9aca19fe695"
server_key_iv: "fd96e48ace3221b2b916e24547634d30"
server_key_version: "encryptionkey@0"
I20260927 05:04:42.579542 11065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:04:42.583173 11080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:42.584656 11065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 05:04:42.584873 11065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea"
format_stamp: "Formatted at 2026-09-27 05:04:42 on dist-test-slave-39f2"
server_key: "47bd2a16c739106ef338f9aca19fe695"
server_key_iv: "fd96e48ace3221b2b916e24547634d30"
server_key_version: "encryptionkey@0"
I20260927 05:04:42.585079 11065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:42.597306 11065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:42.601950 11065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:42.602281 11065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:42.621397 11065 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:45891
I20260927 05:04:42.621398 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:45891 every 8 connection(s)
I20260927 05:04:42.622939 11065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:42.625425 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11065
I20260927 05:04:42.625643 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:42.626111 21157 external_mini_cluster.cc:1428] Setting key 6d97003ced133a44d912d3868bb5ccbf
I20260927 05:04:42.626992 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:42.640595 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:42.627953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55895 (local address 127.20.169.126:45891)
0927 05:04:42.628723 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:42.628739 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:42.628788 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:04:42.630545 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:42.630571 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:42.631082 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:42.631506 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:42.632003 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:42.634061 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:42.635362 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:42.636338 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:42.636515 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:42.637782 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:42.637792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:42.637815 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:42.638575 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:42.638662 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:42.638699 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:42.638821 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:42.639047 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 05:04:42.639328 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":328,"threads_started":1}
I20260927 05:04:42.641736 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea: Bootstrap starting.
I20260927 05:04:42.644599 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:42.645438 11132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:42.646705 11132 log.cc:824] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:42.649799 11132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea: No bootstrap required, opened a new log
I20260927 05:04:42.655839 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:42.649741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47483 (local address 127.20.169.126:45891)
0927 05:04:42.649920 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:42.649929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:42.650420 (+ 491us) server_negotiation.cc:407] Connection header received
0927 05:04:42.650635 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:42.650645 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:42.650701 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:42.650801 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:42.651297 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:42.652050 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:42.653402 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:42.653905 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:42.654068 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:42.655115 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:42.655120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:42.655123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:42.655144 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:42.655179 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:42.655193 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:42.655293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:42.655432 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 05:04:42.655517 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:04:42.657866 11132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45891 } }
I20260927 05:04:42.658234 11132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:42.658339 11132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d2b1c1f3f744c5b118c16a1c78e9ea, State: Initialized, Role: FOLLOWER
I20260927 05:04:42.658890 11132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45891 } }
I20260927 05:04:42.659132 11132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:42.659274 11132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:42.659471 11132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:42.661998 11132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45891 } }
I20260927 05:04:42.662468 11132 leader_election.cc:302] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08d2b1c1f3f744c5b118c16a1c78e9ea; no voters:
I20260927 05:04:42.662992 11132 leader_election.cc:288] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:42.663151 11142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:42.663798 11142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [term 1 LEADER]: Becoming Leader. State: Replica: 08d2b1c1f3f744c5b118c16a1c78e9ea, State: Running, Role: LEADER
I20260927 05:04:42.664263 11142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45891 } }
I20260927 05:04:42.665057 11132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:42.667363 11143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45891 } } }
I20260927 05:04:42.667372 11144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08d2b1c1f3f744c5b118c16a1c78e9ea. Latest consensus state: current_term: 1 leader_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d2b1c1f3f744c5b118c16a1c78e9ea" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 45891 } } }
I20260927 05:04:42.667843 11144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:42.672477 11143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:42.675125 11149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:42.680444 11149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:42.689584 11149 catalog_manager.cc:1347] Generated new cluster ID: 7e0919f9439a47d2b3ebd7ccf031f010
I20260927 05:04:42.689697 11149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:42.720723 11149 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:42.722332 11149 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:42.730703 11149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08d2b1c1f3f744c5b118c16a1c78e9ea: Generated new TSK 0
I20260927 05:04:42.731540 11149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:42.738133 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11065
W20260927 05:04:42.739378 11127 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55895
W20260927 05:04:42.740127 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11065: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:42.740269 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11065
2026-09-27T05:04:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:42 dist-test-slave-39f2 krb5kdc[11163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:42 dist-test-slave-39f2 krb5kdc[11163](info): set up 2 sockets
Sep 27 05:04:42 dist-test-slave-39f2 krb5kdc[11163](info): commencing operation
krb5kdc: starting...
W20260927 05:04:43.878288 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:44 dist-test-slave-39f2 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:44.136444 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33547
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:33547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:44.461809 11176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:44.462076 11176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:44.462167 11176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:44.469120 11176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:44.469228 11176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:44.469338 11176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:44.469444 11176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:44.469529 11176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:44.479470 11176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:33547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:33547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:44.480787 11176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:44.482183 11176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:44.494203 11181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:44.494303 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:44.494174 11182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:44.495353 11176 server_base.cc:1034] running on GCE node
I20260927 05:04:44.496199 11176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:44.497654 11176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:44.498859 11176 hybrid_clock.cc:648] HybridClock initialized: now 1790485484498814 us; error 34 us; skew 500 ppm
I20260927 05:04:44.499204 11176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:44 dist-test-slave-39f2 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485484, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:44.502266 11176 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:44.503754 11176 webserver.cc:460] Webserver started at http://127.20.169.126:44109/ using document root <none> and password file <none>
I20260927 05:04:44.504441 11176 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:44.504592 11176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:44.504969 11176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:44.507892 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "77c7cb44f70e4ee08051e4229928f23d"
format_stamp: "Formatted at 2026-09-27 05:04:44 on dist-test-slave-39f2"
server_key: "20009069775bc7bef47248822be5141a"
server_key_iv: "c805b538b4b5e229b049da953a0a7e95"
server_key_version: "encryptionkey@0"
I20260927 05:04:44.508639 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "77c7cb44f70e4ee08051e4229928f23d"
format_stamp: "Formatted at 2026-09-27 05:04:44 on dist-test-slave-39f2"
server_key: "20009069775bc7bef47248822be5141a"
server_key_iv: "c805b538b4b5e229b049da953a0a7e95"
server_key_version: "encryptionkey@0"
I20260927 05:04:44.513633 11176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 05:04:44.516978 11191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:44.518302 11176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:04:44.518473 11176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "77c7cb44f70e4ee08051e4229928f23d"
format_stamp: "Formatted at 2026-09-27 05:04:44 on dist-test-slave-39f2"
server_key: "20009069775bc7bef47248822be5141a"
server_key_iv: "c805b538b4b5e229b049da953a0a7e95"
server_key_version: "encryptionkey@0"
I20260927 05:04:44.518666 11176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:44.524251 11176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:44.528085 11176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:44.528376 11176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:44.546764 11176 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:33547
I20260927 05:04:44.546788 11244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:33547 every 8 connection(s)
I20260927 05:04:44.548125 11176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:44.552450 11245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:44.552480 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11176
I20260927 05:04:44.552645 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:44.553274 21157 external_mini_cluster.cc:1428] Setting key 0a2aba435d71ed94de5862a801cf3e30
I20260927 05:04:44.561029 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:44.555105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36915 (local address 127.20.169.126:33547)
0927 05:04:44.555958 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:44.555971 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:44.556010 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:04:44.557702 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:44.557723 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:44.558047 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:44.558451 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:44.559440 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:44.559449 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:44.559462 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:44.559479 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:44.559536 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:44.559557 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:44.559793 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:44.559935 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 05:04:44.560101 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":357,"threads_started":1}
I20260927 05:04:44.564529 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:44.566834 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d: Bootstrap starting.
I20260927 05:04:44.570869 11245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:44.571260 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:44.568657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38853 (local address 127.20.169.126:33547)
0927 05:04:44.568867 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:44.568875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:44.569340 (+ 465us) server_negotiation.cc:407] Connection header received
0927 05:04:44.569525 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:44.569531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:44.569587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:44.569683 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:44.570632 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:44.570637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:44.570642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:04:44.570658 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:44.570703 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:04:44.570716 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:44.570843 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:44.570936 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 05:04:44.570993 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:04:44.572221 11245 log.cc:824] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:44.575016 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d: No bootstrap required, opened a new log
I20260927 05:04:44.583395 11245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c7cb44f70e4ee08051e4229928f23d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33547 } }
I20260927 05:04:44.583737 11245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:04:44.583823 11245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c7cb44f70e4ee08051e4229928f23d, State: Initialized, Role: FOLLOWER
I20260927 05:04:44.584352 11245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c7cb44f70e4ee08051e4229928f23d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33547 } }
I20260927 05:04:44.584578 11245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:04:44.584678 11245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:04:44.584847 11245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:44.587194 11245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c7cb44f70e4ee08051e4229928f23d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33547 } }
I20260927 05:04:44.587838 11245 leader_election.cc:302] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c7cb44f70e4ee08051e4229928f23d; no voters:
I20260927 05:04:44.588478 11245 leader_election.cc:288] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:04:44.588621 11255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:44.589401 11255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [term 1 LEADER]: Becoming Leader. State: Replica: 77c7cb44f70e4ee08051e4229928f23d, State: Running, Role: LEADER
I20260927 05:04:44.589931 11255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c7cb44f70e4ee08051e4229928f23d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33547 } }
I20260927 05:04:44.590607 11245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:44.592602 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c7cb44f70e4ee08051e4229928f23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c7cb44f70e4ee08051e4229928f23d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33547 } } }
I20260927 05:04:44.593017 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:44.593490 11257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c7cb44f70e4ee08051e4229928f23d. Latest consensus state: current_term: 1 leader_uuid: "77c7cb44f70e4ee08051e4229928f23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c7cb44f70e4ee08051e4229928f23d" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 33547 } } }
I20260927 05:04:44.593848 11257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:44.595276 11260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:44.601516 11260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:44.610953 11260 catalog_manager.cc:1347] Generated new cluster ID: 830a2ea391274701ae29f941fde0f4ea
I20260927 05:04:44.611088 11260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:44.626741 11260 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:44.627278 11260 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:44.632911 11260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c7cb44f70e4ee08051e4229928f23d: Generated new TSK 0
I20260927 05:04:44.633828 11260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:44.648339 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11176
W20260927 05:04:44.649571 11240 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36915
W20260927 05:04:44.650240 21157 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11176: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:04:44.650395 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11176
2026-09-27T05:04:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1909 ms)
[----------] 18 tests from AuthTokenIssuingTest (35708 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:44 dist-test-slave-39f2 krb5kdc[11276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:44 dist-test-slave-39f2 krb5kdc[11276](info): set up 2 sockets
Sep 27 05:04:44 dist-test-slave-39f2 krb5kdc[11276](info): commencing operation
krb5kdc: starting...
W20260927 05:04:45.765172 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:45 dist-test-slave-39f2 krb5kdc[11276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:46.057282 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:46827
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:46827,127.20.169.125:36229,127.20.169.124:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:46.385257 11289 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:04:46.385637 11289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:46.385780 11289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:46.385928 11289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:46.393294 11289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:46.393383 11289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:46.393479 11289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:46.393558 11289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:46.393671 11289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:46.403703 11289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:46827,127.20.169.125:36229,127.20.169.124:43313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:46.405133 11289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:46.406473 11289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:46.418946 11294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:46.418893 11297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:46.418911 11295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:46.420100 11289 server_base.cc:1034] running on GCE node
I20260927 05:04:46.420954 11289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:46.422545 11289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:46.423842 11289 hybrid_clock.cc:648] HybridClock initialized: now 1790485486423761 us; error 49 us; skew 500 ppm
I20260927 05:04:46.424208 11289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:46 dist-test-slave-39f2 krb5kdc[11276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:46.427627 11289 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:46.429251 11289 webserver.cc:460] Webserver started at http://127.20.169.126:41005/ using document root <none> and password file <none>
I20260927 05:04:46.429994 11289 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:46.430141 11289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:46.430495 11289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:46.433668 11289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "88e3f16a680e49b3adc09c596725413a"
format_stamp: "Formatted at 2026-09-27 05:04:46 on dist-test-slave-39f2"
server_key: "b7ba885a9c29a8873f869b788e1aee16"
server_key_iv: "c0056ce4ed4d99cd017e25519f63fb93"
server_key_version: "encryptionkey@0"
I20260927 05:04:46.434571 11289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "88e3f16a680e49b3adc09c596725413a"
format_stamp: "Formatted at 2026-09-27 05:04:46 on dist-test-slave-39f2"
server_key: "b7ba885a9c29a8873f869b788e1aee16"
server_key_iv: "c0056ce4ed4d99cd017e25519f63fb93"
server_key_version: "encryptionkey@0"
I20260927 05:04:46.439985 11289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:04:46.443547 11304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:46.444947 11289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:04:46.445160 11289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "88e3f16a680e49b3adc09c596725413a"
format_stamp: "Formatted at 2026-09-27 05:04:46 on dist-test-slave-39f2"
server_key: "b7ba885a9c29a8873f869b788e1aee16"
server_key_iv: "c0056ce4ed4d99cd017e25519f63fb93"
server_key_version: "encryptionkey@0"
I20260927 05:04:46.445396 11289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:46.459548 11289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:46.463665 11289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:46.463950 11289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:46.483152 11289 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:46827
I20260927 05:04:46.483163 11356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:46827 every 8 connection(s)
I20260927 05:04:46.484575 11289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:46.488364 11357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:46.493988 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11289
I20260927 05:04:46.494199 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:46.495036 21157 external_mini_cluster.cc:1428] Setting key 9d90a270b60382ad15acb152a430c43c
I20260927 05:04:46.495245 11357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:46 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:46.520155 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.499134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.20.169.126:46827)
0927 05:04:46.500017 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:46.500034 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:46.500657 (+ 623us) server_negotiation.cc:407] Connection header received
0927 05:04:46.502829 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:46.502839 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:46.503135 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:46.503412 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:46.504587 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:46.505870 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:46.508112 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:46.508655 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:46.508792 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:46.512780 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:46.512863 (+ 83us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:46.512884 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:46.512915 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:46.515860 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:46.516601 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:46.516612 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:46.516620 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:46.516770 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:46.517228 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:46.517236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:46.517240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:46.517771 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:46.518015 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:46.518733 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 05:04:46.519056 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":628,"thread_start_us":368,"threads_started":1}
I20260927 05:04:46.520267 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.499134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:46827 (local address 127.0.0.1:59892)
0927 05:04:46.500467 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.500516 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:46.502447 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:46.503457 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:46.503463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:46.504028 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:46.504372 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:46.504388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:46.506005 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:46.506012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:46.507968 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:46.507984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:46.508920 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:46.508930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:46.509169 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:46.509858 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:46.509891 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:46.512552 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:46.516016 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:46.516028 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:46.516040 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:46.516462 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:46.516942 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:46.516948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:46.516952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:46.517114 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:46.517993 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:46.518004 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:46.518302 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 05:04:46.518607 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 05:04:46.519273 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":90,"spinlock_wait_cycles":42752,"thread_start_us":818,"threads_started":1}
WARNING: no policy specified for kudu/127.20.169.125@KRBTEST.COM; defaulting to no policy
I20260927 05:04:46.523201 11357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:46.524493 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.524058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56796)
0927 05:04:46.524230 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.524386 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 05:04:46.524886 11307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.125@KRBTEST.COM" created.
I20260927 05:04:46.526562 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.526215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56810)
0927 05:04:46.526363 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.526445 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:04:46.526746 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36229: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab.
Entry for principal kudu/127.20.169.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab.
I20260927 05:04:46.556694 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:36229
--webserver_interface=127.20.169.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab
--principal=kudu/127.20.169.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:46827,127.20.169.125:36229,127.20.169.124:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:04:46.576188 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } attempt: 1
I20260927 05:04:46.577630 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.577027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56816)
0927 05:04:46.577272 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.577433 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:04:46.578892 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.578411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56822)
0927 05:04:46.578603 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.578703 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 05:04:46.579128 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36229: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
I20260927 05:04:46.647415 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } attempt: 2
I20260927 05:04:46.648782 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.648176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56824)
0927 05:04:46.648452 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.648585 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:04:46.649996 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.649543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56838)
0927 05:04:46.649721 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.649827 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:04:46.650197 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36229: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
I20260927 05:04:46.741463 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } attempt: 3
I20260927 05:04:46.742676 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.742169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56848)
0927 05:04:46.742399 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.742511 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 05:04:46.743882 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.743379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56860)
0927 05:04:46.743570 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.743672 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 05:04:46.744148 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36229: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
I20260927 05:04:46.887451 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } attempt: 4
I20260927 05:04:46.888754 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.888160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56864)
0927 05:04:46.888477 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.888586 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 05:04:46.889851 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:46.889459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56868)
0927 05:04:46.889606 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:46.889699 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:04:46.890161 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36229: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36229: connect: Connection refused (error 111)
W20260927 05:04:46.895495 11364 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:04:46.895888 11364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:46.896013 11364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:46.896152 11364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:46.903527 11364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:46.903643 11364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:46.903765 11364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:46.903851 11364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:46.903959 11364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:46.913933 11364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:46827,127.20.169.125:36229,127.20.169.124:43313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab
--principal=kudu/127.20.169.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.169.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:46.915225 11364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:46.916527 11364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:46.928289 11369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:46.928696 11372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:46.928736 11370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:46.929167 11364 server_base.cc:1034] running on GCE node
I20260927 05:04:46.930022 11364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:46.931402 11364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:46.932667 11364 hybrid_clock.cc:648] HybridClock initialized: now 1790485486932593 us; error 44 us; skew 500 ppm
I20260927 05:04:46.933051 11364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:46 dist-test-slave-39f2 krb5kdc[11276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:46.936280 11364 init.cc:377] Logged in from keytab as kudu/127.20.169.125@KRBTEST.COM (short username kudu)
I20260927 05:04:46.937753 11364 webserver.cc:460] Webserver started at http://127.20.169.125:39769/ using document root <none> and password file <none>
I20260927 05:04:46.938423 11364 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:46.938509 11364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:46.938769 11364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:46.941761 11364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data/instance:
uuid: "c4d51ef956ee492182348a45e2e9a808"
format_stamp: "Formatted at 2026-09-27 05:04:46 on dist-test-slave-39f2"
server_key: "061cd78b68fab0fd63cd6fcde1f8065a"
server_key_iv: "d7ba98a7430149d257b8eddae484f06f"
server_key_version: "encryptionkey@0"
I20260927 05:04:46.942534 11364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal/instance:
uuid: "c4d51ef956ee492182348a45e2e9a808"
format_stamp: "Formatted at 2026-09-27 05:04:46 on dist-test-slave-39f2"
server_key: "061cd78b68fab0fd63cd6fcde1f8065a"
server_key_iv: "d7ba98a7430149d257b8eddae484f06f"
server_key_version: "encryptionkey@0"
I20260927 05:04:46.947916 11364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:04:46.951534 11379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:46.953109 11364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:04:46.953248 11364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal
uuid: "c4d51ef956ee492182348a45e2e9a808"
format_stamp: "Formatted at 2026-09-27 05:04:46 on dist-test-slave-39f2"
server_key: "061cd78b68fab0fd63cd6fcde1f8065a"
server_key_iv: "d7ba98a7430149d257b8eddae484f06f"
server_key_version: "encryptionkey@0"
I20260927 05:04:46.953433 11364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:46.974599 11364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:46.979360 11364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:46.979586 11364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:46.999122 11364 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.125:36229
I20260927 05:04:46.999148 11431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.125:36229 every 8 connection(s)
I20260927 05:04:47.000484 11364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
I20260927 05:04:47.003522 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11364
I20260927 05:04:47.003698 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-1/wal/instance
I20260927 05:04:47.004149 21157 external_mini_cluster.cc:1428] Setting key 2c36fda142d09ad749e745e7cbd22c70
I20260927 05:04:47.004526 11432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:47.010967 11432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.124@KRBTEST.COM" created.
I20260927 05:04:47.033660 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.013632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.20.169.126:46827)
0927 05:04:47.013893 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.013902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.015107 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 05:04:47.016959 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.016968 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.017062 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.017194 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.018882 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.020062 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.021977 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.022783 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.022919 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.026837 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.026846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.026852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.026894 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.030227 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.030958 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.030963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.030965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.031100 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.031639 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.031646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.031651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.032100 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.032216 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.033265 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.033441 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:04:47.035358 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.014121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:46827 (local address 127.0.0.1:59908)
0927 05:04:47.014917 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.014974 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.016654 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.017465 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.017474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.018068 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.018681 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.018700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.020260 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.020266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.021813 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.021828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.023030 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.023036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.023269 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.024014 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.024037 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.026590 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.030364 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.030370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.030383 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.030805 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.031204 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.031207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.031209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.031468 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.032250 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.032259 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.032801 (+ 542us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.033167 (+ 366us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.033747 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":356,"threads_started":1}
I20260927 05:04:47.037114 11432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:04:47.054044 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.037986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56872)
0927 05:04:47.038185 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.038209 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.038389 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.039297 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.039301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.039653 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.039794 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.039802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.041800 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.041804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.042677 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.042688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.043868 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.043872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.044026 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.044626 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.044643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.046929 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.051547 (+ 4618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.051551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.051554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.051953 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.052388 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.052391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.052394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.052516 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.053314 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.053318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.053450 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.053574 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.053775 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 05:04:47.054561 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.038013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.20.169.125:36229)
0927 05:04:47.038719 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.038728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.038760 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:04:47.038815 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.038825 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.039104 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.039259 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.039969 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.041659 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.042797 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.043711 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.043868 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.047147 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.047163 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.047183 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.047223 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.051414 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.052089 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.052100 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.052107 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.052275 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.052620 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.052628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.052634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.053167 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.053385 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.053724 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.054013 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":493,"thread_start_us":318,"threads_started":1}
I20260927 05:04:47.057335 11432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:47.058527 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.058000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58466)
0927 05:04:47.058163 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.058370 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 05:04:47.058804 11381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.060411 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.059995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58480)
0927 05:04:47.060179 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.060277 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:04:47.060642 11432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab.
Entry for principal kudu/127.20.169.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab.
I20260927 05:04:47.071650 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:43313
--webserver_interface=127.20.169.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab
--principal=kudu/127.20.169.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:46827,127.20.169.125:36229,127.20.169.124:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:04:47.109942 11432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 1
I20260927 05:04:47.111323 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.110677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58488)
0927 05:04:47.110940 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.111078 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:04:47.112450 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.112022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58492)
0927 05:04:47.112201 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.112305 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:04:47.112685 11432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.180954 11432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 2
I20260927 05:04:47.182327 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.181672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58508)
0927 05:04:47.181949 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.182089 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 05:04:47.183490 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.183029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58520)
0927 05:04:47.183227 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.183338 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 05:04:47.183741 11432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.189411 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } attempt: 5
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:04:47.202705 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.190257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.20.169.125:36229)
0927 05:04:47.190446 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.190454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.190474 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:04:47.190683 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.190688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.190761 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.190868 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.191654 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.192355 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.193609 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.194081 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.194250 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.197615 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.197620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.197623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.197653 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.200137 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.200741 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.200748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.200752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.200932 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.201344 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.201349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.201352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.201620 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.201787 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.202163 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.202385 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:04:47.202705 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.190103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56884)
0927 05:04:47.190348 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.190370 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.190508 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.190911 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.190916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.191318 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.191502 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.191511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.192489 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.192494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.193465 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.193477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.194243 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.194256 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.194479 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.195102 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.195121 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.197431 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.200268 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.200275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.200282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.200620 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.201042 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.201046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.201049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.201221 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.201741 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.201746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.201908 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.202046 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.202348 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 05:04:47.203956 11357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:47.205122 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.204684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58524)
0927 05:04:47.204915 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.204994 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 05:04:47.206110 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.205804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58528)
0927 05:04:47.205945 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.206011 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:04:47.206406 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.257640 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 1
I20260927 05:04:47.258935 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.258469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58542)
0927 05:04:47.258672 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.258768 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:04:47.260031 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.259730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58550)
0927 05:04:47.259858 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.259922 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:04:47.260353 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.274997 11432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 3
I20260927 05:04:47.276204 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.275624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58552)
0927 05:04:47.275888 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.276006 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:04:47.277433 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.276951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58554)
0927 05:04:47.277205 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.277295 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260927 05:04:47.277705 11432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.328603 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 2
I20260927 05:04:47.330062 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.329450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58558)
0927 05:04:47.329713 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.329823 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:04:47.331224 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.330822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58570)
0927 05:04:47.331026 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.331104 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:04:47.331517 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
W20260927 05:04:47.407229 11439 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:04:47.407598 11439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:47.407725 11439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:47.407862 11439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:47.414878 11439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:47.414997 11439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:47.415095 11439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:47.415228 11439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:47.415323 11439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:47.420917 11432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 4
I20260927 05:04:47.422103 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.421548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58574)
0927 05:04:47.421800 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.421905 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:04:47.423102 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.422736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58588)
0927 05:04:47.422882 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.422970 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:04:47.423305 11432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
I20260927 05:04:47.426270 11439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:46827,127.20.169.125:36229,127.20.169.124:43313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab
--principal=kudu/127.20.169.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.169.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:47.427521 11439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:47.428722 11439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 05:04:47.437712 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 3
I20260927 05:04:47.438956 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.438336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58598)
0927 05:04:47.438650 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.438755 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260927 05:04:47.439914 11447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:47.440289 11444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:47.440400 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.439818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58612)
0927 05:04:47.440148 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.440275 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260927 05:04:47.440645 11357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:43313: Network error: Client connection negotiation failed: client connection to 127.20.169.124:43313: connect: Connection refused (error 111)
W20260927 05:04:47.440838 11445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:47.441494 11439 server_base.cc:1034] running on GCE node
I20260927 05:04:47.442229 11439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:47.443665 11439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:47.444938 11439 hybrid_clock.cc:648] HybridClock initialized: now 1790485487444868 us; error 35 us; skew 500 ppm
I20260927 05:04:47.445269 11439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485487, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:47.448526 11439 init.cc:377] Logged in from keytab as kudu/127.20.169.124@KRBTEST.COM (short username kudu)
I20260927 05:04:47.450205 11439 webserver.cc:460] Webserver started at http://127.20.169.124:42235/ using document root <none> and password file <none>
I20260927 05:04:47.450905 11439 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:47.451037 11439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:47.451370 11439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:47.454373 11439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data/instance:
uuid: "72894142732b45269c34fd36c0465ebe"
format_stamp: "Formatted at 2026-09-27 05:04:47 on dist-test-slave-39f2"
server_key: "5e1d5b2be769ef332fdb220f8a36b218"
server_key_iv: "9a8988195e41bc747b75f3a08b251973"
server_key_version: "encryptionkey@0"
I20260927 05:04:47.455153 11439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal/instance:
uuid: "72894142732b45269c34fd36c0465ebe"
format_stamp: "Formatted at 2026-09-27 05:04:47 on dist-test-slave-39f2"
server_key: "5e1d5b2be769ef332fdb220f8a36b218"
server_key_iv: "9a8988195e41bc747b75f3a08b251973"
server_key_version: "encryptionkey@0"
I20260927 05:04:47.460997 11439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:04:47.464563 11454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:47.466120 11439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:04:47.466323 11439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal
uuid: "72894142732b45269c34fd36c0465ebe"
format_stamp: "Formatted at 2026-09-27 05:04:47 on dist-test-slave-39f2"
server_key: "5e1d5b2be769ef332fdb220f8a36b218"
server_key_iv: "9a8988195e41bc747b75f3a08b251973"
server_key_version: "encryptionkey@0"
I20260927 05:04:47.466547 11439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:47.484601 11439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:47.489100 11439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:47.489367 11439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:47.507602 11439 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.124:43313
I20260927 05:04:47.507622 11505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.124:43313 every 8 connection(s)
I20260927 05:04:47.508988 11439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
I20260927 05:04:47.512579 11506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:47.518949 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11439
I20260927 05:04:47.519124 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485126081201-21157-0/minicluster-data/master-2/wal/instance
I20260927 05:04:47.519620 21157 external_mini_cluster.cc:1428] Setting key 74377101cd43c51905f10825a01c9832
I20260927 05:04:47.518890 11506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485487, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:04:47.539016 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.521771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.20.169.126:46827)
0927 05:04:47.521989 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.521996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.522705 (+ 709us) server_negotiation.cc:407] Connection header received
0927 05:04:47.524601 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.524605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.524668 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.524748 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.526169 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.526892 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.528557 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.529219 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.529310 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.533265 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.533271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.533274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.533334 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.535909 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.536685 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.536692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.536696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.536903 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.537334 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.537339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.537342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.537612 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.537784 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.538408 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.538635 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:04:47.540189 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.521482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.20.169.124:43313)
0927 05:04:47.522323 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.522338 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.522385 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:04:47.524326 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.524353 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.524642 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.524879 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.526205 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.527605 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.529073 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.529801 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.529942 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.534288 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.534296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.534306 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.534339 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.537160 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.537810 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.537817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.537819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.537946 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.538330 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.538334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.538336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.538708 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.539024 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.539483 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.539706 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":417,"threads_started":1}
I20260927 05:04:47.540210 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.521830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:46827 (local address 127.0.0.1:59924)
0927 05:04:47.522542 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.522582 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.524326 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.524879 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.524886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.525630 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.525995 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.526012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.527056 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.527061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.528417 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.528429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.529440 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.529447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.529657 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.530567 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.530616 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.533066 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.536057 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.536067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.536079 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.536564 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.537012 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.537018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.537021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.537217 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.537764 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.537772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.538077 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.538335 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.538816 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":334,"threads_started":1}
I20260927 05:04:47.541944 11506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485487, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:47.556108 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.542889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36229 (local address 127.0.0.1:56896)
0927 05:04:47.543059 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.543074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.543198 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.543531 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.543535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.543989 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.544177 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.544188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.545309 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.545313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.546314 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.546328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.547020 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.547026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.547227 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.547801 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.547818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.550214 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.553728 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.553734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.553738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.554103 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.554492 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.554497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.554500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.554645 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.555158 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.555163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.555322 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.555466 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.555742 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:04:47.556138 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.542804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.20.169.125:36229)
0927 05:04:47.542999 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.543005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.543142 (+ 137us) server_negotiation.cc:407] Connection header received
0927 05:04:47.543330 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.543342 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.543409 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.543510 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.544309 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.545176 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.546470 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.546877 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.546972 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.550479 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.550486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.550491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.550532 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.553594 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.554229 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.554236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.554239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.554379 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.554755 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.554761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.554764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.555036 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.555194 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.555557 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.555762 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:04:47.557639 11506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:47.559711 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.543807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.20.169.126:46827)
0927 05:04:47.543990 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.543995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.544012 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:04:47.544123 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.544131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.544199 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.544289 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.546140 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.547009 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.548176 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.548797 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.549027 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.553680 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.553685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.553688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.553718 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.556540 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.557459 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.557462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.557465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.557590 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.558038 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.558042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.558044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.558454 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.558738 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.559174 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.559403 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485487, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:04:47.579118 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.558512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.20.169.124:43313)
0927 05:04:47.558696 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.558703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.558724 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:04:47.558927 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.558933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.559014 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.559133 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.560151 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.561064 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.562561 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.562973 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.563078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.566488 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.566493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.566496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.566524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.576022 (+ 9498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.576930 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.576936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.576939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.577097 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.577573 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.577581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.577585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.578016 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.578150 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.578576 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.578742 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:04:47.579381 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.558334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58636)
0927 05:04:47.558622 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.558645 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.558778 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.559148 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.559154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.559698 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.560000 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.560011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.561220 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.561224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.562390 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.562409 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.563132 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.563139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.563369 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.563960 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.563976 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.566293 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.576245 (+ 9952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.576252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.576257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.576714 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.577223 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.577228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.577230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.577417 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.578149 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.578154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.578316 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.578494 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.578942 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":120}
I20260927 05:04:47.584174 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.561648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.20.169.125:36229)
0927 05:04:47.561854 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.561859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.561876 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:04:47.561924 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.561927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.561986 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.562073 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.563547 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.564363 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.565843 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.566541 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.566630 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.578149 (+ 11519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.578155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.578159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.578198 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.581680 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.582298 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.582303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.582307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.582494 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.582842 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.582847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.582850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.583183 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.583453 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.583624 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.583831 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:04:47.585565 11506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: Bootstrap starting.
I20260927 05:04:47.589015 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:47.589315 11506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:47.590569 11506 log.cc:824] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:47.593231 11506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: No bootstrap required, opened a new log
I20260927 05:04:47.592499 11332 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "88e3f16a680e49b3adc09c596725413a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59932
I20260927 05:04:47.600091 11506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.600366 11506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72894142732b45269c34fd36c0465ebe, State: Initialized, Role: FOLLOWER
I20260927 05:04:47.600870 11506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.601785 11512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.602169 11512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.602708 11506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:04:47.613117 11523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:04:47.613279 11523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:04:47.617934 11357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 4
I20260927 05:04:47.610347 11439 master_runner.cc:186] Error getting master registration for 127.20.169.126:46827: OK, instance_id { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" instance_seqno: 1790485486477578 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:04:47.621709 11439 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:04:47.636623 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.618626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58644)
0927 05:04:47.618986 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.619007 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.619116 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.619903 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.619907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.620414 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.620639 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.620653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.622118 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.622124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.623408 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.623423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.624511 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.624519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.624776 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.625549 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.625595 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.628696 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.633525 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.633533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.633537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.634078 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.634588 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.634595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.634598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.634774 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.635562 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.635567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.635724 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.635928 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.636216 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260927 05:04:47.636888 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.619299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.20.169.124:43313)
0927 05:04:47.619454 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.619460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.619476 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:04:47.619559 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.619565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.619634 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.619721 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.620854 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.621968 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.623618 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.624333 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.624500 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.628955 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.628969 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.628973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.629034 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.633365 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.634214 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.634223 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.634241 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.634425 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.634896 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.634902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.634905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.635411 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.635548 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.636111 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.636382 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:04:47.642006 11357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: Bootstrap starting.
I20260927 05:04:47.645638 11357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:47.646836 11357 log.cc:824] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:47.650058 11357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: No bootstrap required, opened a new log
I20260927 05:04:47.655977 11357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.656347 11357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e3f16a680e49b3adc09c596725413a, State: Initialized, Role: FOLLOWER
I20260927 05:04:47.656765 11357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.657768 11526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.658236 11526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.658864 11357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:47.659025 11332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 05:04:47.659294 11332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:47.663511 11332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
W20260927 05:04:47.670514 11537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:04:47.670675 11537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:04:47.671149 11332 leader_election.cc:288] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [CANDIDATE]: Term 1 election: Requested vote from peers c4d51ef956ee492182348a45e2e9a808 (127.20.169.125:36229), 72894142732b45269c34fd36c0465ebe (127.20.169.124:43313)
I20260927 05:04:47.672432 11481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e3f16a680e49b3adc09c596725413a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "72894142732b45269c34fd36c0465ebe"
I20260927 05:04:47.673097 11481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:47.677102 11481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e3f16a680e49b3adc09c596725413a in term 1.
I20260927 05:04:47.678128 11306 leader_election.cc:302] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72894142732b45269c34fd36c0465ebe, 88e3f16a680e49b3adc09c596725413a; no voters:
I20260927 05:04:47.680670 11526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:47.689169 11526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [term 1 LEADER]: Becoming Leader. State: Replica: 88e3f16a680e49b3adc09c596725413a, State: Running, Role: LEADER
I20260927 05:04:47.690284 11526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.669922 11289 master_runner.cc:186] Error getting master registration for 127.20.169.125:36229: OK, instance_id { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" instance_seqno: 1790485486993552 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:04:47.691146 11289 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:04:47.695426 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e3f16a680e49b3adc09c596725413a. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.695875 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:47.702885 11547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:47.713810 11547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 05:04:47.716763 11310 tablet.cc:2369] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:47.722501 11432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } attempt: 5
I20260927 05:04:47.724952 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.682651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.20.169.125:36229)
0927 05:04:47.683079 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.683086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.695164 (+ 12078us) server_negotiation.cc:407] Connection header received
0927 05:04:47.695417 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.695424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.695528 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.695665 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.699868 (+ 4203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.701251 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.706211 (+ 4960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.707032 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.707171 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.712217 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.712225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.712232 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.712288 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.718370 (+ 6082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.719370 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.719376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.719383 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.719569 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.720094 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.720101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.720105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.720527 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.720835 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.722221 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.723552 (+ 1331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260927 05:04:47.733237 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.682429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.20.169.126:46827)
0927 05:04:47.682911 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.682919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.695953 (+ 13034us) server_negotiation.cc:407] Connection header received
0927 05:04:47.696287 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.696297 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.696401 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.696491 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.699492 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.701115 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.704355 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.705158 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.705331 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.709461 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.709468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.709473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.709526 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.716307 (+ 6781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.726574 (+ 10267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.726584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.726591 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.726820 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.728742 (+ 1922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.728752 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.728758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.729338 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.729670 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.732235 (+ 2565us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.732610 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260927 05:04:47.739470 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.698847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.20.169.124:43313)
0927 05:04:47.699086 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.699093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.713593 (+ 14500us) server_negotiation.cc:407] Connection header received
0927 05:04:47.713848 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.713855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.713949 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.714060 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.716195 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.717478 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.719803 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.720599 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.720725 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.727142 (+ 6417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.727155 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.727165 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.727232 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.732298 (+ 5066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.733279 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.733287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.733290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.733586 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.734251 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.734261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.734268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.734782 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.735149 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.738522 (+ 3373us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.738832 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260927 05:04:47.741396 11460 tablet.cc:2369] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:04:47 dist-test-slave-39f2 krb5kdc[11276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485486, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:04:47.775256 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.723195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:43313 (local address 127.0.0.1:58662)
0927 05:04:47.723689 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:47.723711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:47.723830 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:47.742410 (+ 18580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:47.742418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:47.743001 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:47.743233 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.743248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.745169 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.745176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.746839 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:47.746857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.752480 (+ 5623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.752488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:47.752820 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.753636 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:47.753666 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:47.756943 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:47.771929 (+ 14986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.771936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.771940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.772545 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.773131 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:47.773137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:47.773142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:47.773325 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:47.774120 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:47.774127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:47.774340 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 05:04:47.774502 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 05:04:47.774843 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348}
I20260927 05:04:47.776507 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:47.724099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.20.169.124:43313)
0927 05:04:47.741992 (+ 17893us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:47.741999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:47.742020 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:04:47.742081 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:47.742087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:47.742169 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:47.743423 (+ 1254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:47.743614 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.744931 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.751463 (+ 6532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:47.752260 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:47.757232 (+ 4972us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:47.757592 (+ 360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:47.757600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:47.757606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:47.757657 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:47.766157 (+ 8500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.772736 (+ 6579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.772744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.772749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.772970 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:47.773495 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:47.773504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:47.773508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:47.773950 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:47.775553 (+ 1603us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:47.775726 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 05:04:47.776016 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17775,"thread_start_us":26041,"threads_started":1}
I20260927 05:04:47.779677 11481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [term 1 FOLLOWER]: Refusing update from remote peer 88e3f16a680e49b3adc09c596725413a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:47.781638 11543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:47.784089 11432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: Bootstrap starting.
I20260927 05:04:47.802052 11432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:47.803972 11432 log.cc:824] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:47.819357 11432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: No bootstrap required, opened a new log
I20260927 05:04:47.833168 11432 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.833766 11432 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d51ef956ee492182348a45e2e9a808, State: Initialized, Role: FOLLOWER
I20260927 05:04:47.834412 11432 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } }
I20260927 05:04:47.839728 11512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e3f16a680e49b3adc09c596725413a. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.840062 11512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.841673 11555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.843140 11549 mvcc.cc:205] Tried to move back new op lower bound from 7333828557925167104 to 7333828557587247104. Current Snapshot: MvccSnapshot[applied={T|T < 7333828557925167104}]
I20260927 05:04:47.844923 11551 mvcc.cc:205] Tried to move back new op lower bound from 7333828557925167104 to 7333828557587247104. Current Snapshot: MvccSnapshot[applied={T|T < 7333828557925167104}]
I20260927 05:04:47.845995 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.846326 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:47.844375 11432 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:47.844766 11406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:47.845155 11407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e3f16a680e49b3adc09c596725413a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4d51ef956ee492182348a45e2e9a808"
I20260927 05:04:47.851384 11406 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [term 1 FOLLOWER]: Refusing update from remote peer 88e3f16a680e49b3adc09c596725413a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:47.855144 11407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e3f16a680e49b3adc09c596725413a in term 1.
I20260927 05:04:47.857318 11526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:47.858232 11555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.859727 11547 catalog_manager.cc:1347] Generated new cluster ID: de3f8bae418e40aba672c328c4d20ef2
I20260927 05:04:47.859853 11547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:47.863809 11512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.864053 11512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.885012 11559 mvcc.cc:205] Tried to move back new op lower bound from 7333828557925167104 to 7333828557587247104. Current Snapshot: MvccSnapshot[applied={T|T < 7333828557925167104}]
I20260927 05:04:47.901386 11555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e3f16a680e49b3adc09c596725413a. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.901710 11555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.900986 11565 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.902236 11565 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:47.902756 11526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88e3f16a680e49b3adc09c596725413a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e3f16a680e49b3adc09c596725413a" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 46827 } } peers { permanent_uuid: "c4d51ef956ee492182348a45e2e9a808" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36229 } } peers { permanent_uuid: "72894142732b45269c34fd36c0465ebe" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 43313 } } }
I20260927 05:04:47.903132 11526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:47.906023 11547 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:47.908216 11547 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:47.925428 11547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e3f16a680e49b3adc09c596725413a: Generated new TSK 0
I20260927 05:04:47.926671 11547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:47.953017 11572 catalog_manager.cc:1259] Loaded cluster ID: de3f8bae418e40aba672c328c4d20ef2
I20260927 05:04:47.953261 11572 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: loading cluster ID for follower catalog manager: success
I20260927 05:04:47.957836 11572 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: acquiring CA information for follower catalog manager: success
I20260927 05:04:47.960764 11572 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4d51ef956ee492182348a45e2e9a808: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:04:48.074891 11436 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.055915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.20.169.125:36229)
0927 05:04:48.056170 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.056176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.072617 (+ 16441us) server_negotiation.cc:407] Connection header received
0927 05:04:48.073114 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.073121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.073224 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.073332 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.074610 (+ 1278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56932: BlockingRecv error: recv got EOF from 127.0.0.1:56932 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 05:04:48.076907 11359 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.049697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.20.169.126:46827)
0927 05:04:48.049913 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.049921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.071962 (+ 22041us) server_negotiation.cc:407] Connection header received
0927 05:04:48.075302 (+ 3340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.075312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.075417 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.075502 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.076641 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59950: BlockingRecv error: recv got EOF from 127.0.0.1:59950 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 05:04:48.080931 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.078226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.20.169.124:43313)
0927 05:04:48.078397 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.078403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.078898 (+ 495us) server_negotiation.cc:407] Connection header received
0927 05:04:48.079144 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.079150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.079240 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.079333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.080643 (+ 1310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58676: BlockingRecv error: recv got EOF from 127.0.0.1:58676 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:04:48.131080 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.113889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.20.169.125:36229)
0927 05:04:48.114095 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.114103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.122004 (+ 7901us) server_negotiation.cc:407] Connection header received
0927 05:04:48.122574 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.122582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.122809 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.122955 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.123710 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.125272 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.127606 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.128448 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.128578 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.129399 (+ 821us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.130136 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.130260 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.130563 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.130673 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 05:04:48.182394 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.168947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.20.169.124:43313)
0927 05:04:48.169141 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.169149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.179953 (+ 10804us) server_negotiation.cc:407] Connection header received
0927 05:04:48.180579 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.180586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.180760 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.180878 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.182071 (+ 1193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58680: BlockingRecv error: recv got EOF from 127.0.0.1:58680 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:04:48.233547 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.217959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.20.169.125:36229)
0927 05:04:48.218185 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.218192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.225922 (+ 7730us) server_negotiation.cc:407] Connection header received
0927 05:04:48.226443 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.226450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.226585 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.226657 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.227427 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.228741 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.231020 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.231814 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.231920 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.232581 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.232762 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.232870 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.233109 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.233204 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 05:04:48.281373 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.268349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.20.169.124:43313)
0927 05:04:48.268557 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.268565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.279269 (+ 10704us) server_negotiation.cc:407] Connection header received
0927 05:04:48.279810 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.279817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.279914 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.279992 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.281080 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58688: BlockingRecv error: recv got EOF from 127.0.0.1:58688 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:04:48.336994 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.318840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.20.169.125:36229)
0927 05:04:48.319049 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.319058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.329472 (+ 10414us) server_negotiation.cc:407] Connection header received
0927 05:04:48.329702 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.329709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.329817 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.329937 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.330694 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.332031 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.334337 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.335094 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.335211 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.335961 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.336120 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.336194 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.336517 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.336614 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 05:04:48.352380 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.350323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.20.169.124:43313)
0927 05:04:48.350579 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.350586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.350615 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:04:48.350990 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.350997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.351114 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.351244 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.352113 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58692: BlockingRecv error: recv got EOF from 127.0.0.1:58692 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 05:04:48.368269 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.360487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.20.169.125:36229)
0927 05:04:48.360721 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.360727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.361290 (+ 563us) server_negotiation.cc:407] Connection header received
0927 05:04:48.361457 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.361463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.361600 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.361733 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.362227 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.363642 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.365712 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.366546 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.366686 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.367207 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.367374 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.367491 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.367685 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.367772 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260927 05:04:48.383050 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.378519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.20.169.124:43313)
0927 05:04:48.378782 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.378788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.381402 (+ 2614us) server_negotiation.cc:407] Connection header received
0927 05:04:48.381627 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.381634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.381747 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.381869 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.382786 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58704: BlockingRecv error: recv got EOF from 127.0.0.1:58704 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 05:04:48.401445 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.394293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.20.169.125:36229)
0927 05:04:48.394505 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.394512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.395052 (+ 540us) server_negotiation.cc:407] Connection header received
0927 05:04:48.395270 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.395276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.395382 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.395497 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.395981 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.397222 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.399145 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.399828 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.399981 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.400520 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.400679 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.400778 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.400999 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.401091 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 05:04:48.410640 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.408387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.20.169.124:43313)
0927 05:04:48.408567 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.408575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.409125 (+ 550us) server_negotiation.cc:407] Connection header received
0927 05:04:48.409273 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.409279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.409384 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.409502 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.410365 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58712: BlockingRecv error: recv got EOF from 127.0.0.1:58712 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:04:48.454289 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.446952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.20.169.125:36229)
0927 05:04:48.447154 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.447160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.447700 (+ 540us) server_negotiation.cc:407] Connection header received
0927 05:04:48.447927 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.447933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.448033 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.448138 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.448657 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.450063 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.452136 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.452793 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.452993 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.453497 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.453649 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.453751 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.453925 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.454027 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 05:04:48.469244 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.466718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.20.169.124:43313)
0927 05:04:48.466941 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.466948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.467568 (+ 620us) server_negotiation.cc:407] Connection header received
0927 05:04:48.467777 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.467784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.467893 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.468006 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.468993 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58728: BlockingRecv error: recv got EOF from 127.0.0.1:58728 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 05:04:48.532642 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.526824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.20.169.125:36229)
0927 05:04:48.527147 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.527156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.527715 (+ 559us) server_negotiation.cc:407] Connection header received
0927 05:04:48.527945 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.527961 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.528036 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.528139 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.528579 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.529413 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.530744 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.531246 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.531403 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.531842 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.531952 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.532052 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.532206 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.532288 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 05:04:48.540753 11509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:48.538855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.20.169.124:43313)
0927 05:04:48.539013 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.539017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.539437 (+ 420us) server_negotiation.cc:407] Connection header received
0927 05:04:48.539641 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.539645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.539715 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.539795 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:48.540545 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58740: BlockingRecv error: recv got EOF from 127.0.0.1:58740 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:04:48.615450 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.608857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.20.169.125:36229)
0927 05:04:48.609235 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.609243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.609630 (+ 387us) server_negotiation.cc:407] Connection header received
0927 05:04:48.609923 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.609928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.610027 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.610154 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.610680 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.611567 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.613245 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.613909 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.614066 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.614534 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.614710 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.614830 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.614994 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.615099 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260927 05:04:48.616092 11523 catalog_manager.cc:1259] Loaded cluster ID: de3f8bae418e40aba672c328c4d20ef2
I20260927 05:04:48.616217 11523 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: loading cluster ID for follower catalog manager: success
I20260927 05:04:48.619416 11523 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: acquiring CA information for follower catalog manager: success
I20260927 05:04:48.622064 11523 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72894142732b45269c34fd36c0465ebe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:04:48.627661 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.621914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.20.169.124:43313)
0927 05:04:48.622148 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.622156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.622722 (+ 566us) server_negotiation.cc:407] Connection header received
0927 05:04:48.622857 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.622861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.622998 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.623118 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.623476 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.624213 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.625650 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.626084 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.626182 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.626692 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.627112 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.627212 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.627311 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.627373 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:04:48.641350 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.633475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.20.169.126:46827)
0927 05:04:48.634038 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.634047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.634784 (+ 737us) server_negotiation.cc:407] Connection header received
0927 05:04:48.635180 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.635189 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.635450 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.635631 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.636494 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.637439 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.639075 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.639494 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.639632 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.640222 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.640612 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.640739 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.640959 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.641038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":628,"threads_started":1}
I20260927 05:04:48.642400 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.635184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.20.169.124:43313)
0927 05:04:48.635369 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.635374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.635918 (+ 544us) server_negotiation.cc:407] Connection header received
0927 05:04:48.636116 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.636124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.636211 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.636316 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.636928 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.637943 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.640334 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.641016 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.641130 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.641628 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.641726 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.641836 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.642031 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.642105 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 05:04:48.642791 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:48.633809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.20.169.125:36229)
0927 05:04:48.634069 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:48.634078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:48.635282 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 05:04:48.635354 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:48.635360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:48.635456 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:48.635548 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:48.636443 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.637662 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.640164 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:48.640826 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:48.640935 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:48.641905 (+ 970us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:48.641998 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:48.642096 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:48.642414 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 05:04:48.642504 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:04:48.646631 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11289
I20260927 05:04:48.647656 11352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:48.756872 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11289
I20260927 05:04:48.779505 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11364
I20260927 05:04:48.780934 11427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:48.906594 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11364
I20260927 05:04:48.928992 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11439
I20260927 05:04:48.930244 11501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:49.038311 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11439
2026-09-27T05:04:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4393 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:04:49 dist-test-slave-39f2 krb5kdc[11676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:04:49 dist-test-slave-39f2 krb5kdc[11676](info): set up 2 sockets
Sep 27 05:04:49 dist-test-slave-39f2 krb5kdc[11676](info): commencing operation
krb5kdc: starting...
W20260927 05:04:50.217259 21157 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/test-user.keytab.
Sep 27 05:04:50 dist-test-slave-39f2 krb5kdc[11676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.169.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
Entry for principal kudu/127.20.169.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab.
I20260927 05:04:50.486334 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:40121
--webserver_interface=127.20.169.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:40121,127.20.169.125:36189,127.20.169.124:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:04:50.814988 11689 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:04:50.815364 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:50.815507 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:50.815632 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:50.823719 11689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:50.823849 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:50.823952 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:50.824049 11689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:50.824136 11689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:50.834331 11689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:40121,127.20.169.125:36189,127.20.169.124:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.126.keytab
--principal=kudu/127.20.169.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.126:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.169.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:50.835717 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:50.837152 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:50.849402 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:50.849280 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:50.849372 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:50.850474 11689 server_base.cc:1034] running on GCE node
I20260927 05:04:50.851231 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:50.852620 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:50.853829 11689 hybrid_clock.cc:648] HybridClock initialized: now 1790485490853760 us; error 42 us; skew 500 ppm
I20260927 05:04:50.854171 11689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:50 dist-test-slave-39f2 krb5kdc[11676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:50.857297 11689 init.cc:377] Logged in from keytab as kudu/127.20.169.126@KRBTEST.COM (short username kudu)
I20260927 05:04:50.858778 11689 webserver.cc:460] Webserver started at http://127.20.169.126:41205/ using document root <none> and password file <none>
I20260927 05:04:50.859449 11689 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:50.859532 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:50.859854 11689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:50.862763 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data/instance:
uuid: "07a4cae59ccf46148938278c35f38064"
format_stamp: "Formatted at 2026-09-27 05:04:50 on dist-test-slave-39f2"
server_key: "aa538627c9c5b5337890198a8070956a"
server_key_iv: "6fb36f303c854698e1405522ac7580d7"
server_key_version: "encryptionkey@0"
I20260927 05:04:50.863539 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal/instance:
uuid: "07a4cae59ccf46148938278c35f38064"
format_stamp: "Formatted at 2026-09-27 05:04:50 on dist-test-slave-39f2"
server_key: "aa538627c9c5b5337890198a8070956a"
server_key_iv: "6fb36f303c854698e1405522ac7580d7"
server_key_version: "encryptionkey@0"
I20260927 05:04:50.868566 11689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:04:50.871973 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:50.873348 11689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:04:50.873535 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal
uuid: "07a4cae59ccf46148938278c35f38064"
format_stamp: "Formatted at 2026-09-27 05:04:50 on dist-test-slave-39f2"
server_key: "aa538627c9c5b5337890198a8070956a"
server_key_iv: "6fb36f303c854698e1405522ac7580d7"
server_key_version: "encryptionkey@0"
I20260927 05:04:50.873772 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:50.892235 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:50.896204 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:50.896487 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:50.915501 11689 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.126:40121
I20260927 05:04:50.915522 11756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.126:40121 every 8 connection(s)
I20260927 05:04:50.916939 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/data/info.pb
I20260927 05:04:50.920531 11757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:50.922359 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11689
I20260927 05:04:50.922554 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-0/wal/instance
I20260927 05:04:50.923116 21157 external_mini_cluster.cc:1428] Setting key 8079ac0de3ef9f1952ba33a0aa5abf40
I20260927 05:04:50.930006 11757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:50 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.169.125@KRBTEST.COM; defaulting to no policy
I20260927 05:04:50.953054 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:50.933815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.20.169.126:40121)
0927 05:04:50.935470 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:50.935482 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:50.935537 (+ 55us) server_negotiation.cc:407] Connection header received
0927 05:04:50.937150 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:50.937155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:50.937395 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:50.937591 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:50.938855 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:50.940048 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:50.941556 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:50.942086 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:50.942290 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:50.945826 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:50.945834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:50.945846 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:50.945873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:50.948683 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:50.949389 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:50.949396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:50.949398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:50.949529 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:50.949917 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:50.949920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:50.949922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:50.950405 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:50.950616 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:50.951174 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 05:04:50.951569 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":1417,"thread_start_us":1205,"threads_started":1}
I20260927 05:04:50.953074 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:50.933815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:40121 (local address 127.0.0.1:52332)
0927 05:04:50.934646 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:50.934697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:50.936719 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:50.937612 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:50.937619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:50.938325 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:50.938648 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:50.938666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:50.940215 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:50.940221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:50.941419 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:50.941431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:50.942320 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:50.942327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:50.942494 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:50.943203 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:50.943224 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:50.945599 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:50.948837 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:50.948848 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:50.948859 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:50.949250 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:50.949635 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:50.949642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:50.949648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:50.949796 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:50.950538 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:50.950550 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:50.950778 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 05:04:50.951040 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 05:04:50.951686 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":164,"thread_start_us":360,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.125@KRBTEST.COM" created.
I20260927 05:04:50.956373 11757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:50.957588 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:50.957181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51112)
0927 05:04:50.957369 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:50.957475 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 05:04:50.957906 11707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
I20260927 05:04:50.959518 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:50.959165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51116)
0927 05:04:50.959318 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:50.959400 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:04:50.959714 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36189: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab.
Entry for principal kudu/127.20.169.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab.
I20260927 05:04:50.985934 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:36189
--webserver_interface=127.20.169.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab
--principal=kudu/127.20.169.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:40121,127.20.169.125:36189,127.20.169.124:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:04:51.008977 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } attempt: 1
I20260927 05:04:51.010120 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.009651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51118)
0927 05:04:51.009847 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.009980 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 05:04:51.011226 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.010886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51124)
0927 05:04:51.011041 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.011106 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:04:51.011413 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36189: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
I20260927 05:04:51.079655 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } attempt: 2
I20260927 05:04:51.081072 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.080480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51130)
0927 05:04:51.080753 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.080924 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:04:51.082226 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.081850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51132)
0927 05:04:51.082027 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.082104 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:04:51.082438 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36189: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
I20260927 05:04:51.173722 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } attempt: 3
I20260927 05:04:51.174947 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.174459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51144)
0927 05:04:51.174676 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.174786 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:04:51.176117 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.175716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51160)
0927 05:04:51.175883 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.175966 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 05:04:51.176331 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36189: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
I20260927 05:04:51.319572 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } attempt: 4
I20260927 05:04:51.320789 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.320210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51162)
0927 05:04:51.320554 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.320646 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260927 05:04:51.320683 11764 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:04:51.321216 11764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:51.321352 11764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:51.321511 11764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 05:04:51.321859 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.321514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51178)
0927 05:04:51.321663 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.321744 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:04:51.322032 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.125:36189: Network error: Client connection negotiation failed: client connection to 127.20.169.125:36189: connect: Connection refused (error 111)
W20260927 05:04:51.328858 11764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:51.328958 11764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:51.329057 11764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:51.329159 11764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:51.329248 11764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:51.339237 11764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:40121,127.20.169.125:36189,127.20.169.124:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.125.keytab
--principal=kudu/127.20.169.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.125:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.169.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:51.340603 11764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:51.342031 11764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:51.354038 11772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:51.353991 11769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:51.354086 11770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:51.360431 11764 server_base.cc:1034] running on GCE node
I20260927 05:04:51.361434 11764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:51.362922 11764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:51.364144 11764 hybrid_clock.cc:648] HybridClock initialized: now 1790485491364074 us; error 40 us; skew 500 ppm
I20260927 05:04:51.364503 11764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:51 dist-test-slave-39f2 krb5kdc[11676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:51.367836 11764 init.cc:377] Logged in from keytab as kudu/127.20.169.125@KRBTEST.COM (short username kudu)
I20260927 05:04:51.369463 11764 webserver.cc:460] Webserver started at http://127.20.169.125:44393/ using document root <none> and password file <none>
I20260927 05:04:51.370186 11764 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:51.370277 11764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:51.370651 11764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:51.373667 11764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data/instance:
uuid: "59eda163cbce4bc396664e847aa03d56"
format_stamp: "Formatted at 2026-09-27 05:04:51 on dist-test-slave-39f2"
server_key: "bb91b266697ec0cfc5363d6acf922aec"
server_key_iv: "f525e93543cd44a7e7886754e56ef2a7"
server_key_version: "encryptionkey@0"
I20260927 05:04:51.374473 11764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal/instance:
uuid: "59eda163cbce4bc396664e847aa03d56"
format_stamp: "Formatted at 2026-09-27 05:04:51 on dist-test-slave-39f2"
server_key: "bb91b266697ec0cfc5363d6acf922aec"
server_key_iv: "f525e93543cd44a7e7886754e56ef2a7"
server_key_version: "encryptionkey@0"
I20260927 05:04:51.379863 11764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:04:51.383432 11779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:51.384789 11764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:04:51.385017 11764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal
uuid: "59eda163cbce4bc396664e847aa03d56"
format_stamp: "Formatted at 2026-09-27 05:04:51 on dist-test-slave-39f2"
server_key: "bb91b266697ec0cfc5363d6acf922aec"
server_key_iv: "f525e93543cd44a7e7886754e56ef2a7"
server_key_version: "encryptionkey@0"
I20260927 05:04:51.385251 11764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:51.400416 11764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:51.404397 11764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:51.404661 11764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:51.423961 11764 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.125:36189
I20260927 05:04:51.423985 11830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.125:36189 every 8 connection(s)
I20260927 05:04:51.425710 11764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/data/info.pb
I20260927 05:04:51.429430 11831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:51.433009 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11764
I20260927 05:04:51.433195 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-1/wal/instance
I20260927 05:04:51.433764 21157 external_mini_cluster.cc:1428] Setting key 91bb984c4354eae5ef1c1740e5b800c6
I20260927 05:04:51.437774 11831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:51 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:51.461472 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.441187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.20.169.126:40121)
0927 05:04:51.441423 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:51.441429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:51.442829 (+ 1400us) server_negotiation.cc:407] Connection header received
0927 05:04:51.444887 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:51.444892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:51.444964 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:51.445061 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:51.447221 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.447929 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.450485 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.450912 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.451011 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:51.455320 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:51.455325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:51.455328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:51.455364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:51.458124 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:51.459048 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:51.459056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:51.459059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:51.459196 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:51.459665 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:51.459670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:51.459674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:51.459966 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:51.460124 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:51.460890 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 05:04:51.461121 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.20.169.124@KRBTEST.COM; defaulting to no policy
I20260927 05:04:51.462579 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.441750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:40121 (local address 127.0.0.1:52334)
0927 05:04:51.442628 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.442680 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:51.444491 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:51.445361 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:51.445375 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:51.446199 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:51.447016 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:51.447045 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.448102 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.448111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:51.450309 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:51.450328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.451154 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.451163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:51.451502 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:51.452269 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:51.452311 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:51.455061 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:51.458281 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:51.458290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:51.458309 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:51.458901 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:51.459313 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:51.459320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:51.459323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:51.459547 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:51.460098 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:51.460106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:51.460490 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 05:04:51.460769 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 05:04:51.461343 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":357,"threads_started":1}
I20260927 05:04:51.464469 11831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.169.124@KRBTEST.COM" created.
Sep 27 05:04:51 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:04:51.479314 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.465462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51186)
0927 05:04:51.465693 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.465718 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:51.465827 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:51.466726 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:51.466732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:51.467181 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:51.467319 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:51.467328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.468715 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.468722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:51.469618 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:51.469630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.470451 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.470456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:51.470641 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:51.471247 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:51.471313 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:51.473374 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:51.476582 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:51.476590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:51.476595 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:51.477060 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:51.477435 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:51.477441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:51.477444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:51.477673 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:51.478339 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:51.478346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:51.478494 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 05:04:51.478639 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 05:04:51.478907 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:04:51.479405 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.465614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.20.169.125:36189)
0927 05:04:51.466303 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:51.466309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:51.466334 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:04:51.466374 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:51.466380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:51.466569 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:51.466735 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:51.467495 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.468583 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.469767 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.470308 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.470443 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:51.473589 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:51.473600 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:51.473613 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:51.473641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:51.476445 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:51.477189 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:51.477199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:51.477203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:51.477328 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:51.477797 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:51.477801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:51.477804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:51.478219 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:51.478428 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:51.478815 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 05:04:51.479026 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":420,"threads_started":1}
I20260927 05:04:51.482549 11831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:51.483716 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.483269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37716)
0927 05:04:51.483432 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.483549 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 05:04:51.483966 11783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.485559 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.485182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37728)
0927 05:04:51.485389 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.485462 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:04:51.485760 11831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.169.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab.
Entry for principal kudu/127.20.169.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab.
I20260927 05:04:51.500298 21157 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
/tmp/dist-test-taskRTYp1d/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:44777
--webserver_interface=127.20.169.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab
--principal=kudu/127.20.169.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.169.84:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.169.126:40121,127.20.169.125:36189,127.20.169.124:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:04:51.535085 11831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 1
I20260927 05:04:51.536356 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.535853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37744)
0927 05:04:51.536063 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.536179 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:04:51.537467 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.537113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37748)
0927 05:04:51.537295 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.537365 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:04:51.537688 11831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.605960 11831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 2
I20260927 05:04:51.607204 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.606699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37764)
0927 05:04:51.606924 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.607033 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 05:04:51.608388 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.608020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37780)
0927 05:04:51.608178 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.608259 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 05:04:51.608641 11831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.621263 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } attempt: 5
Sep 27 05:04:51 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:04:51.634868 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.622007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51194)
0927 05:04:51.622191 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.622215 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:51.622358 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:51.622752 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:51.622759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:51.623175 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:51.623357 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:51.623367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.624365 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.624370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:51.625304 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:51.625316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.626087 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.626093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:51.626322 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:51.627035 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:51.627056 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:51.629176 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:51.632376 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:51.632381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:51.632385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:51.632756 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:51.633172 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:51.633177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:51.633181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:51.633335 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:51.633889 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:51.633894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:51.634058 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 05:04:51.634201 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 05:04:51.634506 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 05:04:51.634867 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.622122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.20.169.125:36189)
0927 05:04:51.622320 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:51.622327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:51.622347 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:04:51.622506 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:51.622512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:51.622605 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:51.622713 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:51.623507 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.624222 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.625484 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:51.625937 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:51.626072 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:51.629375 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:51.629380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:51.629383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:51.629416 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:51.632200 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:51.632906 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:51.632913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:51.632919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:51.633054 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:51.633455 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:51.633459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:51.633464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:51.633750 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:51.633914 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:51.634316 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 05:04:51.634539 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:04:51.636436 11757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:51.637485 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.637100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37788)
0927 05:04:51.637268 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.637354 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 05:04:51.638356 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.638035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37794)
0927 05:04:51.638167 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.638249 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:04:51.638579 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.689782 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 1
I20260927 05:04:51.691061 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.690437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37798)
0927 05:04:51.690682 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.690817 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 05:04:51.692262 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.691828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37806)
0927 05:04:51.692016 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.692119 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 05:04:51.692521 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.699878 11831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 3
I20260927 05:04:51.701085 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.700527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37808)
0927 05:04:51.700759 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.700939 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 05:04:51.702170 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.701751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37824)
0927 05:04:51.701987 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.702057 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 05:04:51.702407 11831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.760785 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 2
I20260927 05:04:51.762202 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.761545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37826)
0927 05:04:51.761825 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.761961 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 05:04:51.763415 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.762963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37836)
0927 05:04:51.763168 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.763269 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 05:04:51.763688 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.845595 11831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 4
I20260927 05:04:51.846855 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.846245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37852)
0927 05:04:51.846525 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.846642 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:04:51.848091 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.847644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37858)
0927 05:04:51.847854 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.847947 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:04:51.848371 11831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
W20260927 05:04:51.853549 11838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:04:51.853915 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:04:51.854007 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:04:51.854189 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:04:51.861775 11838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:04:51.861861 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:04:51.861972 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:04:51.862056 11838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:04:51.862164 11838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:04:51.869884 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 3
I20260927 05:04:51.871264 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.870658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37868)
0927 05:04:51.870906 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.871043 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 05:04:51.872924 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.872405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37880)
0927 05:04:51.872682 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:51.872779 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260927 05:04:51.873188 11757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.169.124:44777: Network error: Client connection negotiation failed: client connection to 127.20.169.124:44777: connect: Connection refused (error 111)
I20260927 05:04:51.875306 11838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.169.84:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.169.126:40121,127.20.169.125:36189,127.20.169.124:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/krb5kdc/kudu_127.20.169.124.keytab
--principal=kudu/127.20.169.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.169.124:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.169.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:04:51.876748 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:04:51.878262 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:04:51.891160 11843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:51.891165 11844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:04:51.891155 11846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:04:51.892771 11838 server_base.cc:1034] running on GCE node
I20260927 05:04:51.893622 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:04:51.895102 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:04:51.896487 11838 hybrid_clock.cc:648] HybridClock initialized: now 1790485491896396 us; error 47 us; skew 500 ppm
I20260927 05:04:51.896935 11838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:04:51 dist-test-slave-39f2 krb5kdc[11676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:04:51.900332 11838 init.cc:377] Logged in from keytab as kudu/127.20.169.124@KRBTEST.COM (short username kudu)
I20260927 05:04:51.901840 11838 webserver.cc:460] Webserver started at http://127.20.169.124:44151/ using document root <none> and password file <none>
I20260927 05:04:51.902526 11838 fs_manager.cc:359] Metadata directory not provided
I20260927 05:04:51.902616 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:04:51.902884 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:04:51.905997 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data/instance:
uuid: "353a57dedcb646bc9202cf261a07811e"
format_stamp: "Formatted at 2026-09-27 05:04:51 on dist-test-slave-39f2"
server_key: "2a3ba41868ea1a40d6ff60df0e597ae9"
server_key_iv: "cd1575f4a19a77fac615674a3ea968cd"
server_key_version: "encryptionkey@0"
I20260927 05:04:51.906883 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal/instance:
uuid: "353a57dedcb646bc9202cf261a07811e"
format_stamp: "Formatted at 2026-09-27 05:04:51 on dist-test-slave-39f2"
server_key: "2a3ba41868ea1a40d6ff60df0e597ae9"
server_key_iv: "cd1575f4a19a77fac615674a3ea968cd"
server_key_version: "encryptionkey@0"
I20260927 05:04:51.912365 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:04:51.916427 11854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:04:51.917971 11838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:04:51.918169 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data,/tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal
uuid: "353a57dedcb646bc9202cf261a07811e"
format_stamp: "Formatted at 2026-09-27 05:04:51 on dist-test-slave-39f2"
server_key: "2a3ba41868ea1a40d6ff60df0e597ae9"
server_key_iv: "cd1575f4a19a77fac615674a3ea968cd"
server_key_version: "encryptionkey@0"
I20260927 05:04:51.918396 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:04:51.955909 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:04:51.960733 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:04:51.961105 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:04:51.982128 11838 rpc_server.cc:307] RPC server started. Bound to: 127.20.169.124:44777
I20260927 05:04:51.982177 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.169.124:44777 every 8 connection(s)
I20260927 05:04:51.983568 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/data/info.pb
I20260927 05:04:51.987399 11906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:04:51.990865 21157 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu as pid 11838
I20260927 05:04:51.991045 21157 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRTYp1d/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485126081201-21157-0/minicluster-data/master-2/wal/instance
I20260927 05:04:51.991580 21157 external_mini_cluster.cc:1428] Setting key 00118e3242c0306afcd54af5247350c3
I20260927 05:04:51.995325 11906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:52.015753 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.999063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.20.169.126:40121)
0927 05:04:51.999631 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:51.999639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.000154 (+ 515us) server_negotiation.cc:407] Connection header received
0927 05:04:52.000361 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.000366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.000445 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.000556 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.001874 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.002821 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.004636 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.005196 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.005306 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.009302 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.009307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.009310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.009377 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.012097 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.012959 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.012967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.012974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.013143 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.013606 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.013612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.013615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.013956 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.014075 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.015038 (+ 963us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.015286 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":291,"threads_started":1}
I20260927 05:04:52.017295 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.999210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.126:40121 (local address 127.0.0.1:52338)
0927 05:04:52.000016 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:52.000048 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:52.000230 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:52.000617 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:52.000624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:52.001386 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:52.001698 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.001710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.003011 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.003015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.004461 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.004486 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.005439 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.005449 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.005672 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.006324 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:52.006358 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:52.009101 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:52.012243 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.012254 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.012268 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.012773 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.013264 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.013270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.013273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.013482 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.014091 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:52.014099 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:52.014563 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 05:04:52.014898 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 05:04:52.015506 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":71,"spinlock_wait_cycles":21888,"thread_start_us":388,"threads_started":1}
I20260927 05:04:52.017326 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:51.993547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.20.169.124:44777)
0927 05:04:51.994369 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:51.994381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:51.994423 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:04:51.996217 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:51.996241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:51.996705 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:51.997204 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:51.998965 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.001149 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.002648 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.003541 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.003693 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.008473 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.008485 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.008503 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.008553 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.012258 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.013070 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.013077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.013080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.013265 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.013708 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.013711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.013713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.014538 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.014883 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.015210 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.015630 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":606,"thread_start_us":398,"threads_started":1}
I20260927 05:04:52.019063 11906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.126@KRBTEST.COM
I20260927 05:04:52.033998 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.019969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.20.169.125:36189)
0927 05:04:52.020138 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.020162 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.020186 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:04:52.020327 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.020332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.020424 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.020530 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.021510 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.022383 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.023806 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.024299 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.024393 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.028123 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.028128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.028132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.028167 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.031160 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.031986 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.031990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.031993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.032132 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.032620 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.032624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.032627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.032983 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.033115 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.033548 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.033713 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:04:52.034118 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.019875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.125:36189 (local address 127.0.0.1:51202)
0927 05:04:52.020036 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:52.020055 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:52.020200 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:52.020566 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:52.020569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:52.021128 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:52.021328 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.021341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.022529 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.022534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.023664 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.023676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.024467 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.024474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.024708 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.025374 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:52.025398 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:52.027923 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:52.031314 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.031323 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.031326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.031861 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.032276 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.032279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.032282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.032501 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.033114 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:52.033119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:52.033288 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 05:04:52.033463 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 05:04:52.033759 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:04:52.035198 11906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:04:52.037170 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.021020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.20.169.126:40121)
0927 05:04:52.021175 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.021179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.021214 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:04:52.021473 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.021479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.021564 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.021668 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.023119 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.023842 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.025783 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.026178 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.026271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.030642 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.030647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.030650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.030680 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.033995 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.034772 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.034778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.034781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.034913 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.035491 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.035496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.035498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.035809 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.036133 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.036499 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.036799 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.124@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.169.125@KRBTEST.COM
I20260927 05:04:52.049703 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.035970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.20.169.124:44777)
0927 05:04:52.036145 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.036153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.036172 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:04:52.036390 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.036397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.036481 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.036605 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.037359 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.038277 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.039816 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.040263 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.040428 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.043595 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.043599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.043603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.043630 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.046795 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.047431 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.047448 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.047451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.047619 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.048088 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.048092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.048094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.048388 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.048538 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.049003 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.049377 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":66}
I20260927 05:04:52.049710 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.035899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37892)
0927 05:04:52.036112 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:52.036130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:52.036221 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:52.036643 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:52.036648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:52.037051 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:52.037191 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.037199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.038425 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.038429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.039660 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.039677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.040486 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.040492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.040682 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.041359 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:52.041377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:52.043386 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:52.046944 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.046951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.046955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.047299 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.047768 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.047772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.047775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.047965 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.048519 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:52.048526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:52.048688 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 05:04:52.048877 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 05:04:52.049185 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 05:04:52.050432 11757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 4
I20260927 05:04:52.056414 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.039065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.20.169.125:36189)
0927 05:04:52.039221 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.039226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.039241 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:04:52.039384 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.039388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.039444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.039522 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.040994 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.041668 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.042817 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.043276 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.043407 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.047984 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.047991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.047999 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.048042 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.051719 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.052381 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.052389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.052392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.052538 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.052996 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.053003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.053007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.053385 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.053673 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.055818 (+ 2145us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.056047 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:04:52.057293 11906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: Bootstrap starting.
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485490, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.126@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:04:52.061928 21157 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:04:52.061950 11906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:52.063076 11906 log.cc:824] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:52.066634 11906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: No bootstrap required, opened a new log
I20260927 05:04:52.068526 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.051077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37898)
0927 05:04:52.051251 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:52.051276 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:52.051385 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:52.051849 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:52.051854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:52.052405 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:52.052661 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.052677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.053892 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.053898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.055521 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.055537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.056283 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.056289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.056493 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.057272 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:52.057297 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:52.059676 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:52.065243 (+ 5567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.065248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.065252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.065878 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.066416 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.066426 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.066431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.066628 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.067445 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:52.067453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:52.067626 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 05:04:52.067822 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 05:04:52.068118 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:04:52.068640 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.051348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.20.169.124:44777)
0927 05:04:52.051545 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.051550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.051567 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:04:52.051614 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.051618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.051685 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.051797 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.052851 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.053735 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.055670 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.056135 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.056278 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.059876 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.059882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.059887 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.059925 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.065090 (+ 5165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.066060 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.066066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.066070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.066243 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.066794 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.066801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.066806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.067261 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.067455 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.067986 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.068230 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:04:52.064330 11732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "07a4cae59ccf46148938278c35f38064"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52344
I20260927 05:04:52.074817 11906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.075152 11906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353a57dedcb646bc9202cf261a07811e, State: Initialized, Role: FOLLOWER
I20260927 05:04:52.075361 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064: Bootstrap starting.
I20260927 05:04:52.075683 11906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.076627 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.077091 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.077919 11906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:52.079502 11757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:52.080729 11757 log.cc:824] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:52.085990 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064: No bootstrap required, opened a new log
I20260927 05:04:52.094926 11757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.095340 11757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a4cae59ccf46148938278c35f38064, State: Initialized, Role: FOLLOWER
I20260927 05:04:52.095799 11757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
W20260927 05:04:52.096608 11925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:04:52.096721 11925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:04:52.096971 11927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.097381 11927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.097965 11757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:52.098059 11732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 05:04:52.098279 11732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:52.102283 11732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.104841 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07a4cae59ccf46148938278c35f38064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "353a57dedcb646bc9202cf261a07811e"
I20260927 05:04:52.105383 11881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:52.094383 11838 master_runner.cc:186] Error getting master registration for 127.20.169.126:40121: OK, instance_id { permanent_uuid: "07a4cae59ccf46148938278c35f38064" instance_seqno: 1790485490910145 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:04:52.106029 11838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:04:52.109390 11881 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07a4cae59ccf46148938278c35f38064 in term 1.
I20260927 05:04:52.110050 11938 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:04:52.110152 11709 leader_election.cc:302] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a4cae59ccf46148938278c35f38064, 353a57dedcb646bc9202cf261a07811e; no voters:
I20260927 05:04:52.110471 11732 leader_election.cc:288] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [CANDIDATE]: Term 1 election: Requested vote from peers 59eda163cbce4bc396664e847aa03d56 (127.20.169.125:36189), 353a57dedcb646bc9202cf261a07811e (127.20.169.124:44777)
I20260927 05:04:52.110898 11927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:04:52.121985 11927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [term 1 LEADER]: Becoming Leader. State: Replica: 07a4cae59ccf46148938278c35f38064, State: Running, Role: LEADER
I20260927 05:04:52.123044 11927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.137566 11939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a4cae59ccf46148938278c35f38064. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.138029 11939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:52.147583 11831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } attempt: 5
W20260927 05:04:52.149641 11711 tablet.cc:2369] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:52.135547 11689 master_runner.cc:186] Error getting master registration for 127.20.169.125:36189: OK, instance_id { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" instance_seqno: 1790485491418388 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:04:52.156265 11689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:04:52.157281 11947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:04:52.172032 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.115391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.20.169.125:36189)
0927 05:04:52.115628 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.115637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.130247 (+ 14610us) server_negotiation.cc:407] Connection header received
0927 05:04:52.130850 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.130860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.131106 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.131216 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.136651 (+ 5435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.137545 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.141779 (+ 4234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.142751 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.142899 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.161601 (+ 18702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.161608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.161613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.161657 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.165430 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.167152 (+ 1722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.167160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.167188 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.167435 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.169540 (+ 2105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.169546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.169549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.169952 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.170518 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.171093 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.171626 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:04:52.173087 11947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:04:52.176469 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.114721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.20.169.126:40121)
0927 05:04:52.114918 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.114924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.138248 (+ 23324us) server_negotiation.cc:407] Connection header received
0927 05:04:52.138703 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.138708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.138835 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.138909 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.152670 (+ 13761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.154182 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.162071 (+ 7889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.163136 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.163264 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.169293 (+ 6029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.169303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.169312 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.169395 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.173109 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.173926 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.173931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.173934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.174103 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.174565 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.174570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.174573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.174975 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.175269 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.175769 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.176030 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 05:04:52 dist-test-slave-39f2 krb5kdc[11676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485491, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.169.125@KRBTEST.COM for kudu/127.20.169.124@KRBTEST.COM
I20260927 05:04:52.190232 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.136773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.20.169.124:44777)
0927 05:04:52.137012 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.137019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.161103 (+ 24084us) server_negotiation.cc:407] Connection header received
0927 05:04:52.163822 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.163829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.163946 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.164037 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.174767 (+ 10730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.175770 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.178605 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.179011 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.179148 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.182392 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.182399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.182403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.182446 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.186902 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.187726 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.187734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.187738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.187902 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.188362 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.188368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.188371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.188712 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.189004 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.189540 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.189810 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 05:04:52.192739 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.148161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.169.124:44777 (local address 127.0.0.1:37918)
0927 05:04:52.148522 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 05:04:52.148542 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:04:52.148649 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:04:52.171982 (+ 23333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:04:52.171987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:04:52.172510 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:04:52.172749 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.172763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.174809 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.174816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.176372 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:04:52.176394 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.177516 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.177523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:04:52.177835 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.178468 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:04:52.178492 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:04:52.181463 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:04:52.188926 (+ 7463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.188931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.188934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.189445 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.190158 (+ 713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:04:52.190163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:04:52.190166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:04:52.190372 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:04:52.191748 (+ 1376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:04:52.191754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:04:52.191910 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 05:04:52.192062 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 05:04:52.192360 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260927 05:04:52.194141 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.149037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.20.169.124:44777)
0927 05:04:52.171477 (+ 22440us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.171483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.171510 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:04:52.171567 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.171572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.171660 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.172985 (+ 1325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.173278 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.174614 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.176608 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.177331 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.178175 (+ 844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.181833 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:04:52.181844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:04:52.181849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:04:52.181941 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:04:52.188723 (+ 6782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.189680 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.189687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.189693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.189954 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:04:52.190859 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:04:52.190865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:04:52.190869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:04:52.191543 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:04:52.193389 (+ 1846us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.193524 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.193738 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22261,"thread_start_us":22717,"threads_started":1}
I20260927 05:04:52.200091 11831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: Bootstrap starting.
I20260927 05:04:52.204047 11831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: Neither blocks nor log segments found. Creating new log.
I20260927 05:04:52.205271 11831 log.cc:824] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: Log is configured to *not* fsync() on all Append() calls
I20260927 05:04:52.206636 11881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [term 1 FOLLOWER]: Refusing update from remote peer 07a4cae59ccf46148938278c35f38064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:52.207639 11939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 05:04:52.213516 11860 tablet.cc:2369] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:04:52.226234 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a4cae59ccf46148938278c35f38064. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.226588 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.228659 11951 mvcc.cc:205] Tried to move back new op lower bound from 7333828576022925312 to 7333828575742992384. Current Snapshot: MvccSnapshot[applied={T|T < 7333828576022925312}]
I20260927 05:04:52.229028 11831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: No bootstrap required, opened a new log
I20260927 05:04:52.234040 11953 mvcc.cc:205] Tried to move back new op lower bound from 7333828576022925312 to 7333828575742992384. Current Snapshot: MvccSnapshot[applied={T|T < 7333828576022925312}]
I20260927 05:04:52.242022 11831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.242507 11831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eda163cbce4bc396664e847aa03d56, State: Initialized, Role: FOLLOWER
I20260927 05:04:52.243176 11831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } }
I20260927 05:04:52.244756 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.245149 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.245307 11927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.245651 11927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:52.246526 11947 catalog_manager.cc:1347] Generated new cluster ID: c305a3dbd23242f4b349702ba8cb539d
I20260927 05:04:52.246636 11947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:04:52.250584 11957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.251009 11957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.252846 11831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:04:52.252904 11806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07a4cae59ccf46148938278c35f38064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59eda163cbce4bc396664e847aa03d56"
I20260927 05:04:52.253350 11806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:04:52.257802 11806 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07a4cae59ccf46148938278c35f38064 in term 1.
I20260927 05:04:52.261354 11805 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [term 1 FOLLOWER]: Refusing update from remote peer 07a4cae59ccf46148938278c35f38064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:04:52.262367 11939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:04:52.290575 11947 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:04:52.292411 11947 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:04:52.304050 11957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a4cae59ccf46148938278c35f38064. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.304363 11957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.304181 11927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.304481 11927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064 [sys.catalog]: This master's current role is: LEADER
I20260927 05:04:52.304698 11966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07a4cae59ccf46148938278c35f38064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a4cae59ccf46148938278c35f38064" member_type: VOTER last_known_addr { host: "127.20.169.126" port: 40121 } } peers { permanent_uuid: "59eda163cbce4bc396664e847aa03d56" member_type: VOTER last_known_addr { host: "127.20.169.125" port: 36189 } } peers { permanent_uuid: "353a57dedcb646bc9202cf261a07811e" member_type: VOTER last_known_addr { host: "127.20.169.124" port: 44777 } } }
I20260927 05:04:52.305135 11966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:04:52.330176 11947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a4cae59ccf46148938278c35f38064: Generated new TSK 0
I20260927 05:04:52.331328 11947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:04:52.340006 11973 catalog_manager.cc:1259] Loaded cluster ID: c305a3dbd23242f4b349702ba8cb539d
I20260927 05:04:52.340121 11973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: loading cluster ID for follower catalog manager: success
I20260927 05:04:52.367660 11973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: acquiring CA information for follower catalog manager: success
I20260927 05:04:52.370780 11973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59eda163cbce4bc396664e847aa03d56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:04:52.472714 11835 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.458320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.20.169.125:36189)
0927 05:04:52.458561 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.458569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.470651 (+ 12082us) server_negotiation.cc:407] Connection header received
0927 05:04:52.471119 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.471126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.471233 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.471315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.472481 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51398: BlockingRecv error: recv got EOF from 127.0.0.1:51398 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 05:04:52.475417 11910 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.467292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.20.169.126:40121)
0927 05:04:52.467530 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.467539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.473387 (+ 5848us) server_negotiation.cc:407] Connection header received
0927 05:04:52.473653 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.473660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.473741 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.473814 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.475188 (+ 1374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49784: BlockingRecv error: recv got EOF from 127.0.0.1:49784 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 05:04:52.478837 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.476518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.20.169.124:44777)
0927 05:04:52.476691 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.476712 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.477290 (+ 578us) server_negotiation.cc:407] Connection header received
0927 05:04:52.477539 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.477546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.477636 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.477705 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.478613 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42234: BlockingRecv error: recv got EOF from 127.0.0.1:42234 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:04:52.530755 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.511625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.20.169.125:36189)
0927 05:04:52.511804 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.511811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.522766 (+ 10955us) server_negotiation.cc:407] Connection header received
0927 05:04:52.523142 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.523148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.523396 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.523563 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.524287 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.525547 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.527678 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.528332 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.528424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.529189 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.529899 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.530046 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.530293 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.530388 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 05:04:52.573494 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.563974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.20.169.124:44777)
0927 05:04:52.564181 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.564187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.571330 (+ 7143us) server_negotiation.cc:407] Connection header received
0927 05:04:52.571700 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.571707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.571850 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.571946 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.573171 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42238: BlockingRecv error: recv got EOF from 127.0.0.1:42238 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:04:52.627319 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.608416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.20.169.125:36189)
0927 05:04:52.608617 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.608624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.619314 (+ 10690us) server_negotiation.cc:407] Connection header received
0927 05:04:52.619922 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.619930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.620059 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.620133 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.620937 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.622263 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.624668 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.625453 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.625561 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.626265 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.626443 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.626539 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.626852 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.626962 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 05:04:52.674024 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.664658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.20.169.124:44777)
0927 05:04:52.664899 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.664907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.671829 (+ 6922us) server_negotiation.cc:407] Connection header received
0927 05:04:52.672355 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.672363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.672472 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.672558 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.673723 (+ 1165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42254: BlockingRecv error: recv got EOF from 127.0.0.1:42254 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:04:52.723654 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.706389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.20.169.125:36189)
0927 05:04:52.706588 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.706596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.715847 (+ 9251us) server_negotiation.cc:407] Connection header received
0927 05:04:52.716253 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.716260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.716400 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.716482 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.717277 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.718656 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.720939 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.721765 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.721883 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.722598 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.722801 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.722887 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.723196 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.723296 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 05:04:52.766669 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.756999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.20.169.124:44777)
0927 05:04:52.757176 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.757182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.764474 (+ 7292us) server_negotiation.cc:407] Connection header received
0927 05:04:52.765006 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.765013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.765154 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.765234 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.766358 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42256: BlockingRecv error: recv got EOF from 127.0.0.1:42256 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:04:52.795722 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.787272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.20.169.125:36189)
0927 05:04:52.787527 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.787534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.788199 (+ 665us) server_negotiation.cc:407] Connection header received
0927 05:04:52.788614 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.788622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.788749 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.788903 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.789856 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.791144 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.793324 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.794187 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.794342 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.794834 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.794999 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.795113 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.795300 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.795401 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260927 05:04:52.807140 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.804283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.20.169.124:44777)
0927 05:04:52.804522 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.804529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.805114 (+ 585us) server_negotiation.cc:407] Connection header received
0927 05:04:52.805402 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.805409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.805504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.805599 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.806681 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42260: BlockingRecv error: recv got EOF from 127.0.0.1:42260 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 05:04:52.828441 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.819698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.20.169.125:36189)
0927 05:04:52.819911 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.819919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.820479 (+ 560us) server_negotiation.cc:407] Connection header received
0927 05:04:52.820875 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.820882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.820985 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.821067 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.822212 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.823430 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.825841 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.826543 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.826677 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.827438 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.827603 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.827678 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.827949 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.828054 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 05:04:52.837952 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.835716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.20.169.124:44777)
0927 05:04:52.835911 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.835916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.836530 (+ 614us) server_negotiation.cc:407] Connection header received
0927 05:04:52.836754 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.836759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.836896 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.837002 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.837740 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42264: BlockingRecv error: recv got EOF from 127.0.0.1:42264 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:04:52.863718 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.857445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.20.169.125:36189)
0927 05:04:52.857679 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.857686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.858060 (+ 374us) server_negotiation.cc:407] Connection header received
0927 05:04:52.858226 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.858233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.858332 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.858476 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.858902 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.860132 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.861592 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.862274 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.862446 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.862887 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.863033 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.863152 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.863285 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.863382 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 05:04:52.871834 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.869874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.20.169.124:44777)
0927 05:04:52.870096 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.870103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.870418 (+ 315us) server_negotiation.cc:407] Connection header received
0927 05:04:52.870557 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.870563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.870670 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.870805 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.871559 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42276: BlockingRecv error: recv got EOF from 127.0.0.1:42276 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:04:52.913105 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.907046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.20.169.125:36189)
0927 05:04:52.907288 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.907295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.907510 (+ 215us) server_negotiation.cc:407] Connection header received
0927 05:04:52.907677 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.907684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.907816 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.907941 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.908305 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.909519 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.910843 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.911635 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.911812 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.912202 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.912349 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.912460 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.912576 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.912662 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 05:04:52.921649 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:52.919376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.20.169.124:44777)
0927 05:04:52.919609 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.919618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.920141 (+ 523us) server_negotiation.cc:407] Connection header received
0927 05:04:52.920354 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.920359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.920442 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.920550 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:52.921402 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42288: BlockingRecv error: recv got EOF from 127.0.0.1:42288 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:04:52.998715 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:52.989186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.20.169.125:36189)
0927 05:04:52.989415 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:52.989420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:52.993414 (+ 3994us) server_negotiation.cc:407] Connection header received
0927 05:04:52.993674 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:52.993678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:52.993761 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:52.993872 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:52.994403 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.995186 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.996653 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:52.997163 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:52.997323 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:52.997925 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:52.998044 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:52.998155 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:52.998317 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 05:04:52.998402 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 05:04:53.008141 11949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:04:53.006289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.20.169.124:44777)
0927 05:04:53.006481 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:53.006488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:53.006847 (+ 359us) server_negotiation.cc:407] Connection header received
0927 05:04:53.006982 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:53.006988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:53.007088 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:53.007212 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:04:53.007915 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42290: BlockingRecv error: recv got EOF from 127.0.0.1:42290 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:04:53.098775 11925 catalog_manager.cc:1259] Loaded cluster ID: c305a3dbd23242f4b349702ba8cb539d
I20260927 05:04:53.098879 11925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: loading cluster ID for follower catalog manager: success
I20260927 05:04:53.101547 11925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: acquiring CA information for follower catalog manager: success
I20260927 05:04:53.103279 11925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 353a57dedcb646bc9202cf261a07811e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:04:53.147802 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:53.142702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.20.169.125:36189)
0927 05:04:53.142899 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:53.142903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:53.143310 (+ 407us) server_negotiation.cc:407] Connection header received
0927 05:04:53.143484 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:53.143488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:53.143567 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:53.143654 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:53.144119 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.144853 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.146198 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.146693 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.146799 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:53.147172 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:53.147321 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:53.147394 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:53.147527 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 05:04:53.147591 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:04:53.160092 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:53.153684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.20.169.124:44777)
0927 05:04:53.153924 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:53.153932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:53.154214 (+ 282us) server_negotiation.cc:407] Connection header received
0927 05:04:53.154502 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:53.154507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:53.154671 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:53.154818 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:53.155278 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.156019 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.157500 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.157960 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.158120 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:53.158700 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:53.159413 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:53.159588 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:53.159677 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 05:04:53.159780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:04:53.174916 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:53.166310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.20.169.125:36189)
0927 05:04:53.166580 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:53.166590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:53.168224 (+ 1634us) server_negotiation.cc:407] Connection header received
0927 05:04:53.168495 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:53.168503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:53.168606 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:53.168675 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:53.169447 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.170514 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.172732 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.173453 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.173563 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:53.174138 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:53.174277 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:53.174385 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:53.174521 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 05:04:53.174601 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:04:53.175024 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:53.167460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.20.169.124:44777)
0927 05:04:53.167674 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:53.167682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:53.168879 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 05:04:53.169144 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:53.169149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:53.169228 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:53.169336 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:53.169935 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.170868 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.172649 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.173327 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.173460 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:53.173985 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:53.174147 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:53.174235 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:53.174687 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 05:04:53.174764 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:04:53.175958 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:04:53.166143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.20.169.126:40121)
0927 05:04:53.169076 (+ 2933us) server_negotiation.cc:206] Beginning negotiation
0927 05:04:53.169081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:04:53.169101 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:04:53.169213 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:04:53.169218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:04:53.169434 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:04:53.169632 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:04:53.170257 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.171206 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.173302 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:04:53.173970 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:04:53.174157 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:04:53.174700 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:04:53.175259 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:04:53.175427 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 05:04:53.175573 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 05:04:53.175658 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2797,"thread_start_us":2691,"threads_started":1}
I20260927 05:04:53.185093 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11689
I20260927 05:04:53.186117 11752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:53.296553 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11689
I20260927 05:04:53.318356 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11764
I20260927 05:04:53.319468 11826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:53.427016 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11764
I20260927 05:04:53.448267 21157 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu pid 11838
I20260927 05:04:53.449430 11901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:04:53.556895 21157 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRTYp1d/build/asan/bin/kudu with pid 11838
2026-09-27T05:04:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4518 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8912 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (367400 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST